Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer

Open access

Abstract

Advances in the quality of whole-slide images have set the stage for the clinical use of digital images in anatomic pathology. Along with advances in computer image analysis, this raises the possibility for computer-assisted diagnostics in pathology to improve histopathologic interpretation and clinical care. To evaluate the potential impact of digital assistance on interpretation of digitized slides, we conducted a multireader multicase study utilizing our deep learning algorithm for the detection of breast cancer metastasis in lymph nodes. Six pathologists reviewed 70 digitized slides from lymph node sections in 2 reader modes, unassisted and assisted, with a wash-out period between sessions. In the assisted mode, the deep learning algorithm was used to identify and outline regions with high likelihood of containing tumor. Algorithm-assisted pathologists demonstrated higher accuracy than either the algorithm or the pathologist alone. In particular, algorithm assistance significantly increased the sensitivity of detection for micrometastases (91% vs. 83%, P=0.02). In addition, average review time per image was significantly shorter with assistance than without assistance for both micrometastases (61 vs. 116 s, P=0.002) and negative images (111 vs. 137 s, P=0.018). Lastly, pathologists were asked to provide a numeric score regarding the difficulty of each image classification. On the basis of this score, pathologists considered the image review of micrometastases to be significantly easier when interpreted with assistance (P=0.0005). Utilizing a proof of concept assistant tool, this study demonstrates the potential of a deep learning algorithm to improve pathologist accuracy and efficiency in a digital pathology workflow.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1097/pas.0000000000001151

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 Schematic of reader study design. Readers review the same images in the same sequence, but with different modes: algorithm assisted, or unassisted. Readers are randomized to one of the 2 assistance “orders.” Each rectangle indicates a set of images; the color of the rectangle indicates the mode (assisted or unassisted), and the number in the rectangle indicates the number of images in that set. Readers reviewed a total of 5 images for familiarization and 70 images for formal review.
FIGURE 1 Schematic of reader study design. Readers review the same images in the same sequence, but with different modes: algorithm assisted, or unassisted. Readers are randomized to one of the 2 assistance “orders.” Each rectangle indicates a set of images; the color of the rectangle indicates the mode (assisted or unassisted), and the number in the rectangle indicates the number of images in that set. Readers reviewed a total of 5 images for familiarization and 70 images for formal review.CC BY-NC-ND · PubMed Central
FIGURE 2 Digital image assistance by presenting algorithm predictions as a direct overlay; without assistance (A) versus with assistance (B). Suspicious regions of interest are highlighted in cyan for high confidence and green for moderate confidence (based on the algorithm predictions). In this image, both the high and moderate confidence regions of interests are confirmed as metastatic tumor as indicated by cytokeratin IHC (C). Another image example is shown in Supplemental Figure 2 (Supplemental Digital Content 1, http://links.lww.com/PAS/A677).
FIGURE 2 Digital image assistance by presenting algorithm predictions as a direct overlay; without assistance (A) versus with assistance (B). Suspicious regions of interest are highlighted in cyan for high confidence and green for moderate confidence (based on the algorithm predictions). In this image, both the high and moderate confidence regions of interests are confirmed as metastatic tumor as indicated by cytokeratin IHC (C). Another image example is shown in Supplemental Figure 2 (Supplemental Digital Content 1, http://links.lww.com/PAS/A677).CC BY-NC-ND · PubMed Central
FIGURE 3 Improved metastasis detection with algorithm assistance. A, Data represents performance across all images by image category and assistance modality. Error bars indicate SE. The performance metric corresponds to corresponds to specificity for negative cases and sensitivity for micrometastases (micromet) and macrometastases (macromet). B, Operating point of individual pathologists with and without assistance for micrometastases and negative cases, overlayed on the receiver operating characteristic curve of the algorithm. AUC indicates area under the curve.
FIGURE 3 Improved metastasis detection with algorithm assistance. A, Data represents performance across all images by image category and assistance modality. Error bars indicate SE. The performance metric corresponds to corresponds to specificity for negative cases and sensitivity for micrometastases (micromet) and macrometastases (macromet). B, Operating point of individual pathologists with and without assistance for micrometastases and negative cases, overlayed on the receiver operating characteristic curve of the algorithm. AUC indicates area under the curve.CC BY-NC-ND · PubMed Central
FIGURE 4 False-negative interpretations and false-positive algorithm predictions. A, Lymph node with >200 dispersed tumor cells but classified as negative by all readers with or without assistance; selected field of view shows the most concentrated focus of tumor cells. Left: without algorithm overlay, middle: with algorithm overlay; LYNA algorithm highlights small areas within this region with moderate confidence. Right: cytokeratin IHC as reference for presence of tumor cells; the final reference standard classification as micrometastasis for this image was reached based on counting >200 tumor cells on IHC. B, Left: stain quality and “bland” morphology led to poor visual contrast between a small macrometastasis and benign lymphoid tissue. Middle: tumor focus was outlined with moderate confidence, albeit incompletely circumscribed. Right: cytokeratin IHC for reference. Notably, despite the region highlighted by the algorithm, the tumor in this section was missed by pathologists both with and without assistance. C, Representative examples from independent cases in which LYNA falsely highlights histiocytes in the sinus (left), a giant cell (middle), and fat necrosis (right).
FIGURE 4 False-negative interpretations and false-positive algorithm predictions. A, Lymph node with >200 dispersed tumor cells but classified as negative by all readers with or without assistance; selected field of view shows the most concentrated focus of tumor cells. Left: without algorithm overlay, middle: with algorithm overlay; LYNA algorithm highlights small areas within this region with moderate confidence. Right: cytokeratin IHC as reference for presence of tumor cells; the final reference standard classification as micrometastasis for this image was reached based on counting >200 tumor cells on IHC. B, Left: stain quality and “bland” morphology led to poor visual contrast between a small macrometastasis and benign lymphoid tissue. Middle: tumor focus was outlined with moderate confidence, albeit incompletely circumscribed. Right: cytokeratin IHC for reference. Notably, despite the region highlighted by the algorithm, the tumor in this section was missed by pathologists both with and without assistance. C, Representative examples from independent cases in which LYNA falsely highlights histiocytes in the sinus (left), a giant cell (middle), and fat necrosis (right).CC BY-NC-ND · PubMed Central
FIGURE 5 Average review time per image decreases with assistance. A, Average review time per image across all pathologists analyzed by category. Black circles are average times with assistance, gray triangles represent average times without assistance. Error bars indicate 95% confidence interval. B, Micrometastasis time of review decreases for nearly all images with assistance. Circles represent average review time for each individual micrometastasis image, averaged across the 6 pathologists by assistance modality. The dashed lines connect the points corresponding to the same image with and without assistance. The 2 images that were not reviewed faster on average with assistance are represented with red dot-dash lines. Vertical lines of the box represent quartiles, and the diamond indicates the average of review time for micrometastases in that modality. Micromet indicates micrometastasis; macromet, macrometastasis.
FIGURE 5 Average review time per image decreases with assistance. A, Average review time per image across all pathologists analyzed by category. Black circles are average times with assistance, gray triangles represent average times without assistance. Error bars indicate 95% confidence interval. B, Micrometastasis time of review decreases for nearly all images with assistance. Circles represent average review time for each individual micrometastasis image, averaged across the 6 pathologists by assistance modality. The dashed lines connect the points corresponding to the same image with and without assistance. The 2 images that were not reviewed faster on average with assistance are represented with red dot-dash lines. Vertical lines of the box represent quartiles, and the diamond indicates the average of review time for micrometastases in that modality. Micromet indicates micrometastasis; macromet, macrometastasis.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Cited by

See all 542

Metadata

Authors
Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., & Stumpe, M. C.
Affiliations
  • Google AI Healthcare
  • Verily Life Sciences LLC (United States) (US)
Corresponding author
  • Steiner, David F.
  • MacDonald, Robert
  • Liu, Yun
  • Truszkowski, Peter
  • Hipp, Jason D.
  • Gammage, Christopher
  • Peng, Lily
  • Stumpe, Martin C.
Venue
American Journal of Surgical Pathology
Published
2018-12
Volume
42
Issue
12
Pages
1636-1646
ISSN
0147-5185 (ISSN-L)
OpenAlex
W2897434820
Cited by
489
References
28
Citations by year
2026: 48 · 2025: 50 · 2024: 67 · 2023: 68 · 2022: 65 · 2021: 84 · 2020: 67 · 2019: 38 · 2018: 2
FWCI
26.91
Type
journal article
Language
en
Open access
Locations
  • The American Journal of Surgical Pathology (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
computer-aided diagnosis, deep learning, micrometastases, digital pathology, whole slide images, review time
MeSH terms
Proof of Concept Study, Deep Learning, Biopsy, Breast Neoplasms, Diagnosis, Computer-Assisted, Female, Humans, Image Interpretation, Computer-Assisted, Lymph Nodes, Lymphatic Metastasis, Pathology, Clinical, Pattern Recognition, Automated, Predictive Value of Tests, Time Factors, Reproducibility of Results, Observer Variation, Workflow, Neoplasm Micrometastasis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer. https://app.cochat.ai/papers/10.1097/pas.0000000000001151
The paper
Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., & Stumpe, M. C. (2018). Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer. American Journal of Surgical Pathology. https://doi.org/10.1097/pas.0000000000001151

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.