MAESTER: Masked Autoencoder Guided Segmentation at Pixel Resolution for Accurate, Self-Supervised Subcellular Structure Recognition
Abstract
Accurate segmentation of cellular images remains an elusive task due to the intrinsic variability in morphology of biological structures. Complete manual segmentation is unfeasible for large datasets, and while supervised methods have been proposed to automate segmentation, they often rely on manually generated ground truths which are especially challenging and time consuming to generate in biology due to the requirement of domain expertise. Furthermore, these methods have limited generalization capacity, requiring additional manual labels to be generated for each dataset and use case. We introduce MAESTER (Masked AutoEncoder guided Segmen'Iation at pixEl Resolution), a self-supervised method for accurate, subcellular structure segmentation at pixel resolution. MAESTER treats segmentation as a representation learning and clustering problem. Specifically, MAESTER learns semantically meaningful token representations of multi-pixel image patches while simultaneously maintaining a sufficiently large field of view for contextual learning. We also develop a cover-and-stride inference strategy to achieve pixel-level subcellular structure segmentation. We evaluated MAESTER on a publicly available volumetric electron microscopy (VEM) dataset of primary mouse pancreatic islets ß cells and achieved up-wards of 29.1 % improvement over state-of-the-art under the same evaluation criteria. Furthermore, our results are competitive against supervised methods trained on the same tasks, closing the gap between self-supervised and supervised approaches. MAESTER shows promise for alleviating the critical bottleneck of ground truth generation for imaging related data analysis and thereby greatly increasing the rate of biological discovery. Code available at https://github.com/bowang-lab/MAESTER.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MedSAM provides a successful paradigm for adapting natural image foundation models to new domains.
“Additionally, MedSAM provides a successful paradigm for adapting natural image foundation models to new domains, which can be further extended to biological image segmentation30, such as cell segmentation in light microscopy images31 and organelle segmentation in electron microscopy images32.”
From Segment anything in medical images (2024) · cites this paper as [32]
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Least squares quantization in PCM
Lloyd, S. (1982) · IEEE Transactions on Information Theory · doi:10.1109/tit.1982.1056489 · Cited by 15,836 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 4× · see citing works
“[6] During the training stage, FOVs of size F × F are randomly sampled from the training image volume which then gets divided into non-overlapping n× n multi-pixel patches.”
BackgroundMethodInfluentialExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · European Conference on Computer Vision · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,803 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · cited 1× · see citing works
“These could include reconstruction, denoising, color augmentation and rotation prediction [11,22,30,34].”
BackgroundSegmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R. A., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,254 · see citing works
Lichtman, J. W., & Conchello, J.-A. (2005) · Nature Methods · doi:10.1038/nmeth817 · Cited by 1,751 · see citing works
An Introduction to Convolutional Neural NetworksPreprint
O’Shea, K., & Nash, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08458 · Cited by 1,463 · see citing works
Light Microscopy Techniques for Live Cell Imaging
Stephens, D. J., & Allan, V. (2003) · Science · doi:10.1126/science.1082160 · Cited by 1,183 · see citing works
Joint Unsupervised Learning of Deep Representations and Image Clusters
Yang, J., Parikh, D., & Batra, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.556 · Cited by 879 · see citing works
Enhanced FIB-SEM systems for large-volume 3D imaging
Xu, C. S., Hayworth, K. J., Lu, Z., Grob, P., Hassan, A. M., García-Cerdán, J. G., Niyogi, K. K., Nogales, E., Weinberg, R. J., & Hess, H. F. (2017) · eLife · doi:10.7554/elife.25916 · Cited by 411 · see citing works
Connectomes across development reveal principles of brain maturation
Witvliet, D., Mulcahy, B., Mitchell, J. K., Meirovitch, Y., Berger, D. R., Wu, Y., Liu, Y., Koh, W. X., Parvathala, R., Holmyard, D., et al. (2021) · Nature · doi:10.1038/s41586-021-03778-8 · Cited by 393 · see citing works
Exploring the third dimension: Volume electron microscopy comes of age
Peddie, C. J., & Collinson, L. (2014) · Micron · doi:10.1016/j.micron.2014.01.009 · Cited by 332 · see citing works
Whole-cell organelle segmentation in volume electron microscopy
Heinrich, L., Bennett, D., Ackerman, D. G., Park, W., Bogovic, J., Eckstein, N., Petruncio, A., Clements, J., Pang, S., Xu, S. C., et al. (2021) · Nature · doi:10.1038/s41586-021-03977-3 · Cited by 278 · see citing works
The long history of hematoxylin
Titford, M. (2005) · Biotechnic & Histochemistry · doi:10.1080/10520290500138372 · Cited by 226 · see citing works
Peddie, C. J., Genoud, C., Kreshuk, A., Meechan, K., Micheva, K. D., Narayan, K., Pape, C., Parton, R. G., Schieber, N. L., Schwab, Y., et al. (2022) · Nature Reviews Methods Primers · doi:10.1038/s43586-022-00131-9 · Cited by 226 · see citing works
Unsupervised segmentation of 3D medical images based on clustering and deep representation learning
Moriya, T., Roth, H. R., Nakamura, S., Oda, H., Nagara, K., Oda, M., & Mori, K. (2018) · SPIE Medical Imaging: Biomedical Applications in Molecular, Structural, and Functional Imaging · doi:10.1117/12.2293414 · Cited by 59 · see citing works
Automatic segmentation of mitochondria and endolysosomes in volumetric electron microscopy data
Mekuč, M. Ž., Bohak, C., Hudoklin, S., Kim, B. H., Romih, R., Kim, M. Y., & Marolt, M. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103693 · Cited by 48 · see citing works
Contrastive Registration for Unsupervised Medical Image SegmentationPreprint
Liu, L., Avilés-Rivero, A. I., & Schönlieb, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.08894 · Cited by 12 · see citing works
Han, H., Dmitrieva, M., Sauer, A., Tam, K. H., & Rittscher, J. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00204 · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913 · see citing works
“In particular, each bounding box was represented by an embedding pair of the top-left corner point and the bottom-right corner point.”
Self-citationSegment Anything in Medical Images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2304.12306 · Cited by 67 · see citing works
Self-citation
- Authors
- Xie, R., Pang, K., Bader, G. D., & Wang, B.
- Affiliations
- University Health Network (CA)
- University of Toronto (CA)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 3292-3301
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386066470
- Cited by
- 17
- References
- 32
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 6 · 2023: 1
- FWCI
- 17.90
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- self-supervised learning, masked autoencoder, pixel-level segmentation, representation learning, clustering, pancreatic beta cells, ground truth generation, cell segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
