CoChat
Sign Up
Cited by 953Cell Image Analysis TechniquesAI in cancer detectionMedical Image Segmentation Techniques

Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning

Open access
Greenwald, Noah F., Miller, Geneva, Moen, Erick, Kong, Alex, Kagel, Adam, Dougherty, Thomas, Fullaway, Christine Camacho, McIntosh, Brianna J. +21 more·2022·Nature Biotechnology
DOI 10.1038/s41587-021-01094-0PMC PMC9010346

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

A principal challenge in the analysis of tissue imaging data is cell segmentation-the task of identifying the precise boundary of every cell in an image. To address this problem we constructed TissueNet, a dataset for training segmentation models that contains more than 1 million manually labeled cells, an order of magnitude more than all previously published segmentation training datasets. We used TissueNet to train Mesmer, a deep-learning-enabled segmentation algorithm. We demonstrated that Mesmer is more accurate than previous methods, generalizes to the full diversity of tissue types and imaging platforms in TissueNet, and achieves human-level performance. Mesmer enabled the automated extraction of key cellular features, such as subcellular localization of protein signal, which was challenging with previous approaches. We then adapted Mesmer to harness cell lineage information in highly multiplexed datasets and used this enhanced version to quantify cell morphology changes during human gestation. All code, data and models are released as a community resource.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41587-021-01094-0

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The human-in-the-loop strategy used to annotate the TissueNet dataset took two human years of crowdsourced work for 14 image categories.

    “This is the strategy used to annotate the TissueNet dataset, which in total took two human years of crowdsourced work for 14 image categories6,17.”

    From Cellpose 2.0: how to train your own model (2022) · cites this paper as [6]

    The ‘A human-in-the-loop approach for constructing TissueNet’ section describes crowdsourced annotation and reports more than 4,000 person-hours, equivalent to nearly two person-years. It describes data from nine organs and six imaging platforms, but does not support the claim of 14 image categories. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • The within-human upper bound is by construction higher than any across-human upper bound because it excludes inconsistencies in segmentation styles between different annotators.

    “Note that the within-human upper bound is by construction higher than any ‘across-human’ upper bound6, because it excludes inconsistencies in segmentation styles between different annotators.”

    From Cellpose 2.0: how to train your own model (2022) · cites this paper as [6]

    The section ‘Mesmer achieves human-level performance for segmentation’ says expert annotators can disagree and reports comparisons between different annotators. The paper does not establish a within-human upper bound or show that it must be higher than any across-human upper bound because of differences in segmentation style. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • The TissueNet dataset consists of 2,601 training and 1,249 test images of six different tissue types collected using fluorescent microscopy on six different platforms, and each image has manual segmentations of the cells and the nuclei.

    “The TissueNet dataset consists of 2,601 training and 1,249 test images of six different tissue types collected using fluorescent microscopy on six different platforms, and each image has manual segmentations of the cells and the nuclei (https://datasets.deepcell.org/)6.”

    From Cellpose 2.0: how to train your own model (2022) · cites this paper as [6]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 64

  1. [1]Giesen C Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat Methods 11, 417– 422 (2014). 24584193
  2. [2]Keren L MIBI-TOF: A multiplexed imaging platform relates cellular phenotypes and tissue structure. Sci Adv 5, eaax5851 (2019). 31633026 PMC6785247
  3. [3]Huang W, Hennrick K & Drew S A colorful future of quantitative pathology: validation of Vectra technology using chromogenic multiplexed immunohistochemistry and prostate tissue microarrays. Hum Pathol 44, 29– 38 (2013). 22944297
  4. [4]Lin J-R Highly multiplexed immunofluorescence imaging of human tissues and tumors using t-CyCIF and conventional optical microscopes. Elife 7, e31657 (2018). 29993362 PMC6075866
  5. [5]Gerdes MJ Highly multiplexed single-cell analysis of formalin-fixed, paraffin-embedded cancer tissue. Proc National Acad Sci 110, 11982– 11987 (2013). PMC3718135 23818604
  6. [7]Chen KH, Boettiger AN, Moffitt JR, Wang S & Zhuang X Spatially resolved, highly multiplexed RNA profiling in single cells. Science 348, aaa6090 (2015). 25858977 PMC4662681
  7. [8]Lee JH Highly Multiplexed Subcellular RNA Sequencing in Situ. Science 343, 1360– 1363 (2014). 24578530 PMC4140943
  8. [9]Moffitt JR Molecular, spatial and functional single-cell profiling of the hypothalamic preoptic region. Science 362, eaau5324 (2018). 30385464 PMC6482113
  9. [11]Lubeck E, Coskun AF, Zhiyentayev T, Ahmad M & Cai L Single-cell in situ RNA profiling by sequential hybridization. Nat Methods 11, 360– 361 (2014). 24681720 PMC4085791
  10. [12]Eng C-HL Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+. Nature 568, 235– 239 (2019). 30911168 PMC6544023
  11. [13]Rozenblatt-Rosen O The Human Tumor Atlas Network: Charting Tumor Transitions across Space and Time at Single-Cell Resolution. Cell 181, 236– 249 (2020). 32302568 PMC7376497
  12. [14]Snyder MP The human body at cellular resolution: the NIH Human Biomolecular Atlas Program. Nature 574, 187– 192 (2019). 31597973 PMC6800388
  13. [15]Regev A The Human Cell Atlas White Paper. Arxiv (2018).
  14. [18]Mescher A Junqueira’s Basic Histology: Text and Atlas. (McGraw Hill, 2013).
  15. [19]McQuin C CellProfiler 3.0: Next-generation image processing for biology. Plos Biol 16, e2005970 (2018). 29969450 PMC6029841
  16. [21]Schneider CA, Rasband WS & Eliceiri KW NIH Image to ImageJ: 25 years of image analysis. Nat Methods 9, 671– 675 (2012). 22930834 PMC5554542
  17. [22]Berg S ilastik: interactive machine learning for (bio)image analysis. Nat Methods 16, 1226– 1232 (2019). 31570887
  18. [23]Chaumont F. de Icy: an open bioimage informatics platform for extended reproducible research. Nat Methods 9, 690– 696 (2012). 22743774
  19. [24]Belevich I, Joensuu M, Kumar D, Vihinen H & Jokitalo E Microscopy Image Browser: A Platform for Segmentation and Analysis of Multidimensional Datasets. Plos Biol 14, e1002340 (2016). 26727152 PMC4699692
  20. [26]Valen DAV Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments. Plos Comput Biol 12, e1005177 (2016). 27814364 PMC5096676
  21. [28]Stringer C, Wang T, Michaelos M & Pachitariu M Cellpose: a generalist algorithm for cellular segmentation. Nat Methods 18, 100– 106 (2021). 33318659
  22. [29]Hollandi R nucleAIzer: A Parameter-free Deep Learning Framework for Nucleus Segmentation Using Image Style Transfer. Cell Syst 10, 453– 458.e6 (2020). 34222682 PMC8247631
  23. [30]Koyuncu CF, Gunesli GN, Cetin-Atalay R & Gunduz-Demir C DeepDistance: A Multi-task Deep Regression Model for Cell Detection in Inverted Microscopy Images. Arxiv (2019). 32438298
  24. [31]Yang L NuSeT: A deep learning tool for reliably separating and analyzing crowded cells. Plos Comput Biol 16, e1008193 (2020). 32925919 PMC7515182
  25. [32]Yu W CCDB:6843, mus musculus, Neuroblastoma. CIL. doi: 10.7295/W9CCDB6843. · DOI
  26. [33]Koyuncu CF, Cetin□Atalay R & Gunduz□Demir C Object□Oriented Segmentation of Cell Nuclei in Fluorescence Microscopy Images. Cytom Part A 93, 1019– 1028 (2018). 30211975
  27. [34]Ljosa V, Sokolnicki KL & Carpenter AE Annotated high-throughput microscopy image sets for validation. Nat Methods 9, 637– 637 (2012). 22743765 PMC3627348
  28. [35]Kumar N A Multi-Organ Nucleus Segmentation Challenge. Ieee T Med Imaging 39, 1380– 1391 (2020). PMC10439521 31647422
  29. [36]Verma R Multi-organ Nuclei Segmentation and Classification Challenge 2020. (2020) doi: 10.13140/rg.2.2.12290.02244/1. · DOI
  30. [37]Moen E Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. bioRxiv 803205 (2019) doi: 10.1101/803205. · DOI
  31. [38]Gamper J PanNuke Dataset Extension, Insights and Baselines. Arxiv (2020).
  32. [39]Bannon D DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes. Nat Methods 18, 43– 45 (2021). 33398191 PMC8759612
  33. [40]Haberl MG CDeep3M—Plug-and-Play cloud-based deep learning for image segmentation. Nat Methods 15, 677– 680 (2018). 30171236 PMC6548193
  34. [41]Ouyang W, Mueller F, Hjelmare M, Lundberg E & Zimmer C ImJoy: an open-source computational platform for the deep learning era. Nat Methods 16, 1199– 1200 (2019). 31780825
  35. [42]Chamier L. von ZeroCostDL4Mic: an open platform to use Deep-Learning in Microscopy. Biorxiv 2020.03.20.000133 (2020) doi: 10.1101/2020.03.20.000133. · DOI
  36. [43]Hughes AJ Quanti.us: a tool for rapid, flexible, crowd-based annotation of images. Nat Methods 15, 587– 590 (2018). 30065368 PMC8863499
  37. [44]Ouyang W, Le T, Xu H & Lundberg E Interactive biomedical segmentation tool powered by deep learning and ImJoy. F1000research 10, 142 (2021).
  38. [45]Wolny A Accurate and versatile 3D segmentation of plant tissues at cellular resolution. Elife 9, e57613 (2020). 32723478 PMC7447435
  39. [50]Zuiderveld K Graphics Gems. Viii Image Process 474– 485 (1994) doi: 10.1016/b978-0-12-336156-1.50061-6. · DOI · Paper page
  40. [54]Fu C-Y, Shvets M & Berg AC RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free. Arxiv (2019).
  41. [55]Schürch CM Coordinated Cellular Neighborhoods Orchestrate Antitumoral Immunity at the Colorectal Cancer Invasive Front. Cell 182, 1341– 1359.e19 (2020). 32763154 PMC7479520
  42. [56]Ali HR Imaging mass cytometry and multiplatform genomics define the phenogenomic landscape of breast cancer. Nat Cancer 1, 163– 175 (2020). 35122013
  43. [57]Gaglia G HSF1 phase transition mediates stress adaptation and cell fate decisions. Nat Cell Biol 22, 151– 158 (2020). 32015439 PMC7135912
  44. [58]Nelson DE Oscillations in NF-κB Signaling Control the Dynamics of Gene Expression. Science 306, 704– 708 (2004). 15499023
  45. [59]Kumar KP, McBride KM, Weaver BK, Dingwall C & Reich NC Regulated Nuclear-Cytoplasmic Localization of Interferon Regulatory Factor 3, a Subunit of Double-Stranded RNA-Activated Factor 1. Mol Cell Biol 20, 4159– 4168 (2000). 10805757 PMC85785
  46. [60]Wolff AC Recommendations for Human Epidermal Growth Factor Receptor 2 Testing in Breast Cancer: American Society of Clinical Oncology/College of American Pathologists Clinical Practice Guideline Update. J Clin Oncol 31, 3997– 4013 (2013). 24101045
  47. [61]Risom T Transition to invasive breast cancer is associated with progressive changes in the structure and composition of tumor stroma. bioRxiv 2021.01.05.425362 (2021) doi: 10.1101/2021.01.05.425362. PMC8792442 35063072 · DOI
  48. [63]Diagnostic Cytology and Its Histopathologic Bases. (J.B. Lippincott Company, 1979).
  49. [64]Erlebacher A Immunology of the Maternal-Fetal Interface. Annual Review of Immunology 31, 387– 411 (2013). 23298207
  50. [65]Garrido-Gomez T Defective decidualization during and after severe preeclampsia reveals a possible maternal contribution to the etiology. Proc National Acad Sci 114, E8468– E8477 (2017). PMC5635883 28923940
  51. [67]Bankhead P QuPath: Open source software for digital pathology image analysis. Sci Rep-uk 7, 16878 (2017). PMC5715110 29203879
  52. [68]Graham S Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal 58, 101563 (2019). 31561183
  53. [70]Kiemen A In situ characterization of the 3D microanatomy of the pancreas and pancreatic cancer at single cell resolution. bioRxiv 2020.12.08.416909 (2020) doi: 10.1101/2020.12.08.416909. · DOI
  54. [71]Cao J Establishment of a morphological atlas of the Caenorhabditis elegans embryo using deep-learning-based 4D segmentation. Nat Commun 11, 6254 (2020). 33288755 PMC7721714
  55. [72]Schulz D Simultaneous Multiplexed Imaging of mRNA and Proteins with Subcellular Resolution in Breast Cancer Tissue Samples by Mass Cytometry. Cell Syst 6, 531 (2018). 29698648 PMC5929910
  56. [73]McKinley ET Optimized multiplex immunofluorescence single-cell analysis reveals tuft cell heterogeneity. Jci Insight 2, e93487 (2017). PMC5453701 28570279
  57. [74]Patel SS The microenvironmental niche in classic Hodgkin lymphoma is enriched for CTLA-4- positive T-cells that are PD-1-negative. Blood 134, 2059– 2069 (2019). 31697809 PMC7218752
  58. [75]Jackson HW The single-cell pathology landscape of breast cancer. Nature 578, 615– 620 (2020). 31959985
  59. [76]Rashid R Highly multiplexed immunofluorescence images and single-cell data of immune markers in tonsil and lung cancer. Sci Data 6, 323 (2019). 31848351 PMC6917801
  60. [77]McCaffrey EF Multiplexed imaging of human tuberculosis granulomas uncovers immunoregulatory features conserved across tissue and blood. bioRxiv 2020.06.08.140426 (2020) doi: 10.1101/2020.06.08.140426. · DOI
  61. [78]Walt S van der scikit-image: image processing in Python. Peerj 2, e453 (2014). 25024921 PMC4081273
  62. [79]Kingma DP & Ba J Adam: A Method for Stochastic Optimization. Arxiv (2014).
  63. [85]Pedregosa F Scikit-learn: Machine Learning in Python. Arxiv (2012).
  64. [88]Hoyer S & Hamman J xarray: N-D labeled Arrays and Datasets in Python. Journal of Open Research Software 1, 10 (2017).

Cited by

Sorted by citations

Authors
Greenwald, N. F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Fullaway, C. C., McIntosh, B. J., Leow, K. X., Schwartz, M. S., Pavelchek, C., Cui, S., Camplisson, I., Bar-Tal, O., Singh, J., Fong, M., Chaudhry, G., Abraham, Z., Moseley, J., Warshawsky, S., Soon, E., Greenbaum, S., Risom, T., Hollmann, T., Bendall, S. C., Keren, L., Graf, W., Angelo, M., & Van Valen, D.
Affiliations
  • Stanford Medicine (US)
  • California Institute of Technology (US)
  • Stanford University (US)
  • Weizmann Institute of Science (IL)
  • Brown University (US)
  • Memorial Sloan Kettering Cancer Center (US)
Corresponding author
Angelo, Michael
ORCID
Venue
Nature Biotechnology
Publisher
Springer Science and Business Media LLC
Published
2021-11-18
Volume
40
Issue
4
Pages
555-565
ISSN
  • 0733-222X (ISSN-L)
  • 1087-0156
  • 1546-1696
DOI
10.1038/s41587-021-01094-0
PMC
PMC9010346
OpenAlex
W3214596602
Cited by
953
References
13
Citations by year
2026: 207 · 2025: 230 · 2024: 241 · 2023: 165 · 2022: 85 · 2021: 24 · 2019: 1
FWCI
58.37
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Biotechnology (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • CaltechAUTHORS (California Institute of Technology) (submittedVersion, cc-by-nc)
Funders
Rita Allen Foundation
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
cell segmentation, tissue imaging, Mesmer, deep learning, subcellular localization, multiplexed imaging, cell lineage, cell morphology
MeSH terms
Deep Learning, Algorithms, Humans, Image Processing, Computer-Assisted, Data Curation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review