A deep generative model of 3D single-cell organization
Open accessPublished version of a preprint
The DOI matches its Crossref record; 37 of the 38 references it cites were found in a registry or an index.
Abstract
We introduce a framework for end-to-end integrative modeling of 3D single-cell multi-channel fluorescent image data of diverse subcellular structures. We employ stacked conditional β -variational autoencoders to first learn a latent representation of cell morphology, and then learn a latent representation of subcellular structure localization which is conditioned on the learned cell morphology. Our model is flexible and can be trained on images of arbitrary subcellular structures and at varying degrees of sparsity and reconstruction fidelity. We train our full model on 3D cell image data and explore design trade-offs in the 2D setting. Once trained, our model can be used to predict plausible locations of structures in cells where these structures were not imaged. The trained model can also be used to quantify the variation in the location of subcellular structures by generating plausible instantiations of each structure in arbitrary cell geometries. We apply our trained model to a small drug perturbation screen to demonstrate its applicability to new data. We show how the latent representations of drugged cells differ from unperturbed cells as expected by on-target effects of the drugs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






![Fig 7 Residual block used in this model. Each layer of our model is a modified residual layer. In the encoder, the layer input, x, is passed through a 4x convolution kernel with a stride of 2, then a 3x convolution kernel with a stride of 1 or a 1x convolution kernel with a subsequent avg-pooling step. These results are summed along the channel dimensions, added, and passed to the next layer. With the decoder, 4x convolution is replaced with transposed convolution, and pooling replaced with linear upsampling. In the case of the conditional model (components with dotted lines) MT, the reference input xr is linearly interpolated to be the same size as the output, and passed through a 1x kernel. The target label is passed through a 1x kernel, and added to each pixel of the output. Spectral weight normalization [36] is utilized at every convolutional or fully-connected operation. In the case of the 3D model the convolutions are three-dimensional, and the 2D model uses two-dimensional convolutions.](https://figures.cochat.ai/paper-figures/947d61dcd6614b7492ee5a8bf8b6d6e6/a-deep-generative-model-of-3d-single-cell-organization-figure-7.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other systematic image-based approaches have catalogued the location of human proteins in several cell types and used the locations of proteins and structures within cells to identify differences in intracellular spatial patterns among cells in distinct states6–12,19,29.”
From Integrated intracellular organization and its variations in human iPS cells (2023) · cites this paper as [29]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
CellProfiler: image analysis software for identifying and quantifying cell phenotypes
Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Guertin, D. A., Chang, J. H., Lindquist, R. A., Moffat, J., et al. (2006) · Genome Biology · doi:10.1186/gb-2006-7-10-r100 · Cited by 5,157
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00917 · Cited by 4,332
Spectral Normalization for Generative Adversarial Networks
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.05957 · Cited by 3,438
(2017) · International Conference on Learning Representations · Cited by 3,248
Deep generative modeling for single-cell transcriptomics
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., & Yosef, N. (2018) · Nature Methods · doi:10.1038/s41592-018-0229-2 · Cited by 2,530
A draft map of the human proteome
Kim, M.-S., Pinto, S. M., Getnet, D., Nirujogi, R. S., Manda, S. S., Chaerkady, R., Madugundu, A. K., Kelkar, D. S., Isserlin, R., Jain, S., et al. (2014) · Nature · doi:10.1038/nature13302 · Cited by 2,198
NIPS 2016 Tutorial: Generative Adversarial Networks
Goodfellow, I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.00160 · Cited by 1,229
Content-aware image restoration: pushing the limits of fluorescence microscopy
Weigert, M., Schmidt, U., Boothe, T., Müller, A., Dibrov, A., Jain, A., Wilhelm, B., Schmidt, D., Broaddus, C., Culley, S., et al. (2018) · Nature Methods · doi:10.1038/s41592-018-0216-7 · Cited by 1,197
In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images
Christiansen, E. M., Yang, S. J., Ando, D. M., Javaherian, A., Skibinski, G., Lipnick, S., Mount, E., O’Neil, A., Shah, K., Lee, A. K., et al. (2018) · Cell · doi:10.1016/j.cell.2018.03.040 · Cited by 635 · cited 1×
“In cell imaging, deep neural networks 61 (DNNs) are generally applied to perform pixel-level tasks, such as object 62 segmentation [12], label-free prediction [13, 14], de-noising and image restoration [15, 16], 63 or cell-level analyses such as predicting cell fates [17], classifying cell cycle status [18], 64 distin…”
MethodDeep learning massively accelerates super-resolution localization microscopy
Ouyang, W., Aristov, A., Lelek, M., Hao, X., & Zimmer, C. (2018) · Nature Biotechnology · doi:10.1038/nbt.4106 · Cited by 570
Van Valen, D. A., Kudo, T., Lane, K. M., Macklin, D. N., Quach, N. T., DeFelice, M. M., Maayan, I., Tanouchi, Y., Ashley, E. A., & Covert, M. W. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005177 · Cited by 557
Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F., & Johnson, G. R. (2018) · Nature Methods · doi:10.1038/s41592-018-0111-2 · Cited by 495
Self-citationBoland, M. V., & Murphy, R. F. (2001) · Bioinformatics · doi:10.1093/bioinformatics/17.12.1213 · Cited by 479
Deep Feature Consistent Variational Autoencoder
Hou, X., Shen, L., Sun, K., & Qiu, G. (2017) · 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2017.131 · Cited by 334
Reconstructing cell cycle and disease progression using deep learning
Eulenberg, P., Köhler, N., Blasi, T., Filby, A., Carpenter, A. E., Rees, P., Theis, F. J., & Wolf, F. A. (2017) · Nature Communications · doi:10.1038/s41467-017-00623-3 · Cited by 280
Ladder Variational Autoencoders
Sønderby, C. K., Raiko, T., Maaløe, L., Sønderby, S. K., & Winther, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02282 · Cited by 237
Multi-domain translation between single-cell imaging and sequencing data using autoencoders
Yang, K. D., Belyaeva, A., Venkatachalapathy, S., Damodaran, K., Katcoff, A., Radhakrishnan, A., Shivashankar, G. V., & Uhler, C. (2021) · Nature Communications · doi:10.1038/s41467-020-20249-2 · Cited by 156
Waisman, A., Greca, A. L., Möbbs, A. M., Scarafia, M. A., Velazque, N. L. S., Neiman, G., Moro, L. N., Luzzani, C., Sevlever, G. E., Guberman, A., et al. (2019) · Stem Cell Reports · doi:10.1016/j.stemcr.2019.02.004 · Cited by 130
Zhao, T., & Murphy, R. F. (2007) · Cytometry Part A · doi:10.1002/cyto.a.20487 · Cited by 125
GANs for Biological Image Synthesis
Osokin, A., Chessel, A., Salas, R. E. C., & Vaggi, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.245 · Cited by 105
Chen, J., Ding, L., Viana, M. P., Lee, H., Sluezwski, M. F., Morris, B., Hendershott, M. C., Yang, R., Mueller, I. A., & Rafelski, S. M. (2018) · doi:10.1101/491035 · Cited by 99
Self-citationLu, A. X., Kraus, O. Z., Cooper, S., & Moses, A. M. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007348 · Cited by 97
PhenoRipper: software for rapidly profiling microscopy images
Rajaram, S., Pavie, B., Wu, L. F., & Altschuler, S. J. (2012) · Nature Methods · doi:10.1038/nmeth.2097 · Cited by 80
CytoGAN: Generative Modeling of Cell Images
Goldsborough, P., Pawlowski, N., Caicedo, J. C., Singh, S., & Carpenter, A. E. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/227645 · Cited by 62
Predicting cell lineages using autoencoders and optimal transport
Yang, K. D., Damodaran, K., Venkatachalapathy, S., Soylemezoglu, A. C., Shivashankar, G. V., & Uhler, C. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007828 · Cited by 61
Image‐derived, three‐dimensional generative models of cellular organization
Peng, T., & Murphy, R. F. (2011) · Cytometry Part A · doi:10.1002/cyto.a.21066 · Cited by 55
Deep Convolutional and Recurrent Neural Networks for Cell Motility Discrimination and Prediction
Kimmel, J. C., Brack, A. S., & Marshall, W. F. (2019) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2019.2919307 · Cited by 50
Location proteomics: a systems approach to subcellular location
Murphy, R. F. (2005) · Biochemical Society Transactions · doi:10.1042/bst0330535 · Cited by 37
Donovan-Maiye, R., Tapia, J.-J., Sullivan, D. P., Faeder, J. R., Murphy, R. F., Dittrich, M., & Zuckerman, D. M. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004611 · Cited by 37
Self-citationJohnson, G. R., Donovan-Maiye, R. M., & Maleckar, M. M. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/238378 · Cited by 36
Self-citationMol2Image: Improved Conditional Flow Models for Molecule to Image Synthesis
Yang, K., Goldman, S., Jin, W., Lu, A. X., Barzilay, R., Jaakkola, T., & Uhler, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00662 · Cited by 33
Generative Modeling with Conditional Autoencoders: Building an Integrated CellPreprint
Johnson, G. R., Donovan-Maiye, R., & Maleckar, M. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00092 · Cited by 24
Self-citationApplications and Challenges of Machine Learning to Enable Realistic Cellular Simulations
Vasan, R., Rowan, M. P., Lee, C. T., Johnson, G. R., Rangamani, P., & Holst, M. (2020) · Frontiers in Physics · doi:10.3389/fphy.2019.00247 · Cited by 11
Self-citationCellOrganizer: Learning and Using Cell Geometries for Spatial Cell Simulations
Majarian, T. D., Cao-Berg, I., Ruan, X., & Murphy, R. F. (2019) · Methods in molecular biology · doi:10.1007/978-1-4939-9102-0_11 · Cited by 10
Research works printed but not linked: 3
- Kingma DP, Welling M. Auto-encoding variational bayes. arXiv preprint arXiv:13126114. 2013;.
- Alemi AA, Poole B, Fischer I, Dillon JV, Saurous RA, Murphy K. Fixing a broken ELBO. arXiv preprint arXiv:171100464. 2017;.
- GR Johnson, 2017, Building a 3D Integrated Cell, bioRxiv
Cited by
Generative artificial intelligence empowers digital twins in drug discovery and clinical trials
Bordukova, M., Makarov, N., Rodriguez-Esteban, R., Schmich, F., & Menden, M. P. (2024) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2023.2273839 · Cited by 235
“For instance, state-of-the-art cell DTs focus on predicting changes of single cellular characteristics, e.g. transcriptome [6–8] or cell morphology [9,10], while a multimodal DT could jointly cover numerous cell characteristics.”
BackgroundGenerative-AI-Driven Human Digital Twin in IoT Healthcare: A Comprehensive Survey
Chen, J., Shi, Y., Yi, C., Du, H., Kang, J., & Niyato, D. T. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3421918 · Cited by 198
Self-supervised deep learning encodes high-resolution features of protein subcellular localization
Kobayashi, H., Cheveralls, K. C., Leonetti, M. D., & Royer, L. A. (2022) · Nature Methods · doi:10.1038/s41592-022-01541-z · Cited by 133
Integrated intracellular organization and its variations in human iPS cells
Viana, M. P., Chen, J., Knijnenburg, T. A., Vasan, R., Yan, C., Arakaki, J. E., Bailey, M., Berry, B., Borensztejn, A., Brown, E. M., et al. (2023) · Nature · doi:10.1038/s41586-022-05563-7 · Cited by 132
“spatial patterns among cells in distinct states 6–12, 19, 29 .”
Self-citationSynthetic data in biomedicine via generative artificial intelligence
van Breugel, B., Liu, T., Oglić, D., & van der Schaar, M. (2024) · Nature Reviews Bioengineering · doi:10.1038/s44222-024-00245-7 · Cited by 149
Metadata
- Authors
- Donovan-Maiye, R. M., Brown, J. M., Chan, C. K., Ding, L., Yan, C., Gaudreault, N., Theriot, J. A., Maleckar, M. M., Knijnenburg, T. A., & Johnson, G. R.
- Affiliations
- Allen Institute for Cell Science (US)
- Howard Hughes Medical Institute (US)
- Corresponding author
- Knijnenburg, Theo A.
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2022-01-18
- Volume
- 18
- Issue
- 1
- Pages
- e1009155
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC8797242
- OpenAlex
- W4205199447
- Cited by
- 44
- References
- 38
- Citations by year
- 2026: 11 · 2025: 8 · 2024: 10 · 2023: 10 · 2022: 5
- FWCI
- 3.65
- Type
- journal article
- Language
- en
- Update notices
- new_version (2022-01-28)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Allen Institute
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Single-cell and spatial transcriptomics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Advanced Fluorescence Microscopy Techniques
- Keywords
- conditional variational autoencoder, cell morphology, latent representations
- MeSH terms
- Cell Nucleus, Cells, Cultured, Humans, Microscopy, Fluorescence, Models, Biological, Computational Biology, Imaging, Three-Dimensional, Intracellular Space, Cell Shape, Induced Pluripotent Stem Cells, Single-Cell Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.