The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer
Open accessAbstract
Advances in the quality of whole-slide images have set the stage for the clinical use of digital images in anatomic pathology. Along with advances in computer image analysis, this raises the possibility for computer-assisted diagnostics in pathology to improve histopathologic interpretation and clinical care. To evaluate the potential impact of digital assistance on interpretation of digitized slides, we conducted a multireader multicase study utilizing our deep learning algorithm for the detection of breast cancer metastasis in lymph nodes. Six pathologists reviewed 70 digitized slides from lymph node sections in 2 reader modes, unassisted and assisted, with a wash-out period between sessions. In the assisted mode, the deep learning algorithm was used to identify and outline regions with high likelihood of containing tumor. Algorithm-assisted pathologists demonstrated higher accuracy than either the algorithm or the pathologist alone. In particular, algorithm assistance significantly increased the sensitivity of detection for micrometastases (91% vs. 83%, P=0.02). In addition, average review time per image was significantly shorter with assistance than without assistance for both micrometastases (61 vs. 116 s, P=0.002) and negative images (111 vs. 137 s, P=0.018). Lastly, pathologists were asked to provide a numeric score regarding the difficulty of each image classification. On the basis of this score, pathologists considered the image review of micrometastases to be significantly easier when interpreted with assistance (P=0.0005). Utilizing a proof of concept assistant tool, this study demonstrates the potential of a deep learning algorithm to improve pathologist accuracy and efficiency in a digital pathology workflow.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Google developed LYmph Node Assistant (LYNA; ref. 46) and reported that pathologists who used the system were able to detect significantly more micrometastases with shorter review time.”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“While several studies have compared clinician performance with machine learning (ML) models in other fields-such as dermatology [26,27], radiology [28] and pathology [29]-”
From Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“Previous Head-to-Head comparisons of clinician and AI performance-such as in dermatology [26,27], radiology [28], and pathology [29]-have provided valuable insights, but they remained confined to image-based or narrowly defined diagnostic tasks.”
From Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches (2025) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Wolff, A. C., Hammond, M. E. H., Schwartz, J. N., Hagerty, K. L., Allred, D. C., Cote, R. J., Dowsett, M., Fitzgibbons, P. L., Hanna, W. M., Langer, A., et al. (2007) · Journal of Clinical Oncology · doi:10.1200/jco.2006.09.2775 · Cited by 4,434
Wolff, A. C., Hammond, M. E. H., Hicks, D. G., Dowsett, M., McShane, L. M., Allison, K. H., Allred, D. C., Bartlett, J. M. S., Bilous, M., Fitzgibbons, P., et al. (2013) · Journal of Clinical Oncology · doi:10.1200/jco.2013.50.9984 · Cited by 4,030
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 1×
“10 15 Although immunohistochemistry (IHC) is often utilized to improve diagnostic sensitivity, such stains are costly and time consuming, and institution-speci fi c practices for the use of IHC vary considerably.”
BackgroundGiuliano, A. E. (2011) · JAMA · doi:10.1001/jama.2011.90 · Cited by 2,942
Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
Litjens, G., Sánchez, C. I., Timofeeva, N., Hermsen, M., Nagtegaal, I., Kovacs, I., Hulsbergen - van de Kaa, C., Bult, P., van Ginneken, B., & van der Laak, J. (2016) · Scientific Reports · doi:10.1038/srep26286 · Cited by 1,069 · cited 1×
“11 In addition, deep learning algorithms are inherently limited to the task for which they have been speci fi cally trained.”
BackgroundUnintended consequences of machine learning in medicine
Cabitza, F., Rasoini, R., & Gensini, G. F. (2017) · JAMA · doi:10.1001/jama.2017.7797 · Cited by 1,066 · cited 1×
“23,24 In our study, pathologists were informed of the differences between the high con fi dence and moderate con fi dence regions, and to expect the possibility of false positives and false negatives in the assistance.”
BackgroundMicrometastases or Isolated Tumor Cells and the Outcome of Breast Cancer
de Boer, M., van Deurzen, C. H. M., van Dijck, J. A. A. M., Borm, G. F., van Diest, P. J., Adang, E. M. M., Nortier, J. W. R., Rutgers, E. J. T., Seynaeve, C., Menke-Pluymers, M. B. E., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0904832 · Cited by 524 · cited 1×
“002 Macrometastasis (19) 39 (25-55) 34 (21-47) 0.”
Cited by
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
“The combination of pathologists and the algorithm led to the best accuracy, and the algorithm markedly sped up the review of slides 35 .”
BackgroundMethodRajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
“The limited number of prospective studies to date include diabetic retinopathy grading [48 – 50], detection of breast cancer metastases in sentinel lymph node biopsies [51, 52], wrist fracture detection [53], colonic polyp detection [28, 54], and detection of congenital cataracts [55].”
BackgroundArtificial intelligence in digital pathology — new tools for diagnosis and precision oncology
Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V., & Madabhushi, A. (2019) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-019-0252-y · Cited by 1,698
Metadata
- Authors
- Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., & Stumpe, M. C.
- Affiliations
- Google AI Healthcare
- Verily Life Sciences LLC (United States) (US)
- Corresponding author
- Steiner, David F.
- MacDonald, Robert
- Liu, Yun
- Truszkowski, Peter
- Hipp, Jason D.
- Gammage, Christopher
- Peng, Lily
- Stumpe, Martin C.
- Venue
- American Journal of Surgical Pathology
- Published
- 2018-12
- Volume
- 42
- Issue
- 12
- Pages
- 1636-1646
- ISSN
- 0147-5185 (ISSN-L)
- PMC
- PMC6257102
- OpenAlex
- W2897434820
- Cited by
- 489
- References
- 28
- Citations by year
- 2026: 48 · 2025: 50 · 2024: 67 · 2023: 68 · 2022: 65 · 2021: 84 · 2020: 67 · 2019: 38 · 2018: 2
- FWCI
- 26.91
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- The American Journal of Surgical Pathology (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- computer-aided diagnosis, deep learning, micrometastases, digital pathology, whole slide images, review time
- MeSH terms
- Proof of Concept Study, Deep Learning, Biopsy, Breast Neoplasms, Diagnosis, Computer-Assisted, Female, Humans, Image Interpretation, Computer-Assisted, Lymph Nodes, Lymphatic Metastasis, Pathology, Clinical, Pattern Recognition, Automated, Predictive Value of Tests, Time Factors, Reproducibility of Results, Observer Variation, Workflow, Neoplasm Micrometastasis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer. https://app.cochat.ai/papers/10.1097/pas.0000000000001151
- The paper
- Steiner, D. F., MacDonald, R., Liu, Y., Truszkowski, P., Hipp, J. D., Gammage, C., Thng, F., Peng, L., & Stumpe, M. C. (2018). Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer. American Journal of Surgical Pathology. https://doi.org/10.1097/pas.0000000000001151
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.