CoChat
Sign Up
Cited by 109Cutaneous Melanoma Detection and ManagementAI in cancer detectionSpectroscopy Techniques in Biomedical and Chemical Research

Biopsy-Free in Vivo Virtual Histology of Skin Using Deep Learning

Open access
Li, Jingxi, Garfinkel, Jason, Zhang, Xiaoran, Wu, Di, Zhang, Yijie, de Haan, Kevin, Wang, Hongda, Liu, Tairan +5 more·2021·Light: Science & Applications
DOI 10.1038/s41377-021-00674-8PMC PMC8602311
Open this paper in CoChat — free accountOpen PDF

Abstract

An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming, often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free, cellular-level resolution, in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool, it requires specialized training because the acquired images are grayscale, lack nuclear features, and are difficult to correlate with tissue pathology. Here, we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution, enabling visualization of the epidermis, dermal-epidermal junction, and superficial dermis layers. The network was trained under an adversarial learning scheme, which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo, label-free RCM images of normal skin structure, basal cell carcinoma, and melanocytic nevi with pigmented melanocytes, demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41377-021-00674-8

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. [1]Muzic JG Incidence and trends of basal cell carcinoma and cutaneous squamous cell carcinoma: a population-based study in Olmsted county, Minnesota, 2000 to 2010 Mayo Clin. Proc. 2017 92 890 898 28522111 PMC5535132
  2. [2]Lim HW The burden of skin disease in the United States J. Am. Acad. Dermatol. 2017 76 958 972.e2 28259441
  3. [3]Grajdeanu IA Use of imaging techniques for melanocytic naevi and basal cell carcinoma in integrative analysis (Review) Exp. Therapeutic Med. 2020 20 78 86 PMC7271701 32508998
  4. [4]Murzaku EC Hayan S Rao BK Methods and rates of dermoscopy usage: a cross-sectional survey of US dermatologists stratified by years in practice J. Am. Acad. Dermatol. 2014 71 393 395 25037790
  5. [5]Kittler H Diagnostic accuracy of dermoscopy Lancet Oncol. 2002 3 159 165 11902502
  6. [6]Koenig K Riemann I High-resolution multiphoton tomography of human skin with subcellular spatial resolution and picosecond time resolution J. Biomed. Opt. 2003 8 432 439 12880349
  7. [7]Heibel HD Hooey L Cockerell CJ A review of noninvasive techniques for skin cancer detection in dermatology Am. J. Clin. Dermatol. 2020 21 513 524 32383142
  8. [8]Rajadhyaksha M In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast J. Invest. Dermatol. 1995 104 946 952 7769264
  9. [9]Serban ED Role of in vivo reflectance confocal microscopy in the analysis of melanocytic lesions Acta Dermatovenerol. Croat. 2018 26 64 67 29782304
  10. [10]Rajadhyaksha M Reflectance confocal microscopy of skin in vivo: from bench to bedside Lasers Surg. Med. 2017 49 7 19 27785781 PMC5575825
  11. [11]Rao BK Diagnostic accuracy of reflectance confocal microscopy for diagnosis of skin lesions: an update Arch. Pathol. Lab. Med. 2019 143 326 329 30295069
  12. [12]Jain M Evaluation of bedside diagnostic accuracy, learning curve, and challenges for a novice reflectance confocal microscopy reader for skin cancer detection in vivo JAMA Dermatol. 2018 154 962 965 29998289 PMC6143027
  13. [13]Gareau DS Feasibility of digitally stained multimodal confocal mosaics to simulate histopathology J. Biomed. Opt. 2009 14 034050 19566342 PMC2929174
  14. [14]Puliatti S Ex vivo fluorescence confocal microscopy: the first application for real-time pathological examination of prostatic tissue BJU Int. 2019 124 469 476 30908852
  15. [15]Haenssle HA Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists Ann. Oncol. 2018 29 1836 1842 29846502
  16. [16]Esteva A Dermatologist-level classification of skin cancer with deep neural networks Nature 2017 542 115 118 28117445 PMC8382232
  17. [18]Xia D Computationally-guided development of a stromal inflammation histologic biomarker in lung squamous cell carcinoma Sci. Rep. 2018 8 3941 29500362 PMC5834457
  18. [19]Abels E Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association J. Pathol. 2019 249 286 294 31355445 PMC6852275
  19. [20]Amgad M Report on computational assessment of tumor infiltrating lymphocytes from the International Immuno-Oncology Biomarker Working Group npj Breast Cancer 2020 6 16 32411818 PMC7217824
  20. [21]Diao JA Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes Nat. Commun. 2021 12 1613 33712588 PMC7955068
  21. [22]Taylor-Weiner A A machine learning approach enables quantitative measurement of liver histology and disease monitoring in NASH Hepatology 2021 74 133 147 33570776 PMC8361999
  22. [23]Kose K Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a Multiscale Encoder-Decoder Network (MED-Net) Med. Image Anal. 2021 67 101841 33142135 PMC7885250
  23. [24]D’Alonzo M Semantic segmentation of reflectance confocal microscopy mosaics of pigmented lesions using weak labels Sci. Rep. 2021 11 3679 33574486 PMC7878861
  24. [25]Wang HD Deep learning enables cross-modality super-resolution in fluorescence microscopy Nat. Methods 2019 16 103 110 30559434 PMC7276094
  25. [26]Wu YC Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning Nat. Methods 2019 16 1323 1331 31686039
  26. [27]Mela CA Liu Y Application of convolutional neural networks towards nuclei segmentation in localization-based super-resolution fluorescence microscopy images BMC Bioinforma. 2021 22 325 PMC8204587 34130628
  27. [28]Jiao YH Computational interference microscopy enabled by deep learning APL Photonics 2021 6 046103 PMC8931864 35308602
  28. [30]Rivenson Y Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning Nat. Biomed. Eng. 2019 3 466 477 31142829
  29. [31]Rivenson Y Emerging advances to transform histopathology using virtual staining BME Front. 2020 2020 9647163 PMC10521663 37849966
  30. [32]Zhang YJ Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue Light. Sci. Appl. 2020 9 78 32411363 PMC7203145
  31. [33]Rivenson Y PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning Light. Sci. Appl. 2019 8 23 30728961 PMC6363787
  32. [34]Borhani N Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy Biomed. Opt. Express 2019 10 1339 1350 30891350 PMC6420275
  33. [35]Goodfellow, I. J. et al. Generative adversarial nets. In Proc. 27th International Conference on Neural Information Processing Systems 2672–2680 (MIT Press, 2014).
  34. [36]de Haan K Deep-learning-based image reconstruction and enhancement in optical microscopy Proc. IEEE 2020 108 30 50
  35. [37]Drezek RA Laser scanning confocal microscopy of cervical tissue before and after application of acetic acid Am. J. Obstet. Gynecol. 2000 182 1135 1139 10819847
  36. [38]Patel YG Confocal reflectance mosaicing of basal cell carcinomas in Mohs surgical skin excisions J. Biomed. Opt. 2007 12 034027 17614735
  37. [39]Gareau DS Confocal mosaicing microscopy in skin excisions: a demonstration of rapid surgical pathology J. Microsc. 2009 233 149 159 19196421 PMC2662487
  38. [41]Ulrich M Peritumoral clefting in basal cell carcinoma: correlation of in vivo reflectance confocal microscopy and routine histology J. Cutan. Pathol. 2011 38 190 195 21039746
  39. [42]Pellacani G Reflectance confocal microscopy made easy: the 4 must-know key features for the diagnosis of melanoma and nonmelanoma skin cancers J. Am. Acad. Dermatol. 2019 81 520 526 30954581
  40. [43]Navarrete-Dechent C Reflectance confocal microscopy terminology glossary for nonmelanocytic skin lesions: a systematic review J. Am. Acad. Dermatol. 2019 80 1414 1427.e3 30529706 PMC6830574
  41. [44]Zhang, M. Y. et al. Weakly paired multi-domain image translation. In Proc. 31st British Machine Vision Conference. Virtual Event (BMVA Press, 2020).
  42. [45]Mackiewicz-Wysocka M Basal cell carcinoma – diagnosis Contemp. Oncol. 2013 17 337 342 PMC3934050 24592119
  43. [46]Lee NR Periadnexal mucin as an additional histopathologic feature of chronic eczematous dermatitis Ann. Dermatol. 2015 27 133 141 25834351 PMC4377401
  44. [47]Ko CJ Visual perception, cognition, and error in dermatologic diagnosis: key cognitive principles J. Am. Acad. Dermatol. 2019 81 1227 1234 30797839
  45. [49]Paolino G Histology of non-melanoma skin cancers: an update Biomedicines 2017 5 71 PMC5744095 29261131
  46. [50]Henke E Nandigama R Ergün S Extracellular matrix in the tumor microenvironment and its impact on cancer therapy Front. Mol. Biosci. 2020 6 160 32118030 PMC7025524
  47. [51]Mugdha, A. C. & Wilson, J. W. Machine learning approach to synthesizing multiphoton microscopic images from reflectance confocal (Conference Presentation). In Proc. SPIE 10851, Photonics in Dermatology and Plastic Surgery 2019 P 1085106 (SPIE, 2019).
  48. [52]Wilson, J. W. & Mugdha, A. C. Progress in synthetic multiphoton contrast for in vivo microscopy of mucosal melanoma. In Proc. Biophotonics Congress: Biomedical Optics 2020 (Optical Society of America, 2020).
  49. [53]de Haan K Deep learning-based transformation of H&E stained tissues into special stains Nat. Commun. 2021 12 4884 34385460 PMC8361203
  50. [54]Balakrishnan G VoxelMorph: a learning framework for deformable medical image registration IEEE Trans. Med. Imaging 2019 38 1788 1800 30716034
  51. [55]Isola, P. et al. Image-to-image translation with conditional adversarial networks. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition 5967–5976 (IEEE, 2017).
  52. [56]Mao, X. D. et al. Least squares generative adversarial networks. In Proc. 2017 IEEE International Conference on Computer Vision 2813–2821 (IEEE, 2017).
  53. [57]Miyato, T. et al. Spectral normalization for generative adversarial networks. In Proc. 6th International Conference on Learning Representations (ICLR, 2018).
  54. [59]Schlemper J Attention gated networks: learning to leverage salient regions in medical images Med. Image Anal. 2019 53 197 207 30802813 PMC7610718
  55. [60]Wong, S. C. et al. Understanding data augmentation for classification: when to warp? In Proc. 2016 International Conference on Digital Image Computing: Techniques and Applications 1–6 (IEEE, 2016).
  56. [62]Giacomelli MG Virtual hematoxylin and eosin transillumination microscopy using Epi-fluorescence imaging PLoS ONE 2016 11 e0159337 27500636 PMC4976978
  57. [63]Carpenter AE CellProfiler: image analysis software for identifying and quantifying cell phenotypes Genome Biol. 2006 7 R100 17076895 PMC1794559

Cited by

Sorted by citations

Authors
Li, J., Garfinkel, J., Zhang, X., Wu, D., Zhang, Y., de Haan, K., Wang, H., Liu, T., Bai, B., Rivenson, Y., Rubinstein, G., Scumpia, P. O., & Ozcan, A.
Affiliations
  • California NanoSystems Institute (US)
  • University of California, Los Angeles (US)
Corresponding author
  • Rubinstein, Gennady
  • Scumpia, Philip O.
ORCID
Venue
Light: Science & Applications
Publisher
Springer Science and Business Media LLC
Published
2021-11-18
Volume
10
Issue
1
Pages
233-233
ISSN
2047-7538 (ISSN-L)
DOI
10.1038/s41377-021-00674-8
PMC
PMC8602311
OpenAlex
W3214082102
Cited by
109
References
7
Citations by year
2026: 14 · 2025: 27 · 2024: 30 · 2023: 19 · 2022: 18 · 2021: 1
FWCI
6.19
Type
journal article
Language
en
Open access
Licence
Locations
  • Light Science & Applications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
National Science Foundation
Topics
  • Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Spectroscopy Techniques in Biomedical and Chemical Research
Keywords
virtual histology, reflectance confocal microscopy, skin tumors, basal cell carcinoma, melanocytic nevi, virtual staining, hematoxylin and eosin, convolutional neural networks, adversarial learning, epidermis, basement membrane zone, non-invasive skin imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review