CoChat
Sign Up
Cited by 3,293AI in cancer detectionRadiomics and Machine Learning in Medical ImagingArtificial Intelligence in Healthcare and Education

Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

Open access
Ehteshami Bejnordi, Babak, Veta, Mitko, Johannes van Diest, Paul, van Ginneken, Bram, Karssemeijer, Nico, Litjens, Geert, van der Laak, Jeroen A. W. M., and the CAMELYON16 Consortium +62 more·2017·JAMA
DOI 10.1001/jama.2017.14585PMC PMC5820737
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Objective: Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. Design, Setting, and Participants: Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Exposures: Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. Main Outcomes and Measures: The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. Results: The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). Conclusions and Relevance: In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.

Integrity signals

Registry

Crossref · 10.1001/jama.2017.14585

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 3 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    At the slide level, the pathologist without time constraint achieved a sensitivity of 93.8% (95% CI, 86.9%-100.0%).

    “At the slide level in task 2, the pathologist WOTC achieved a sensitivity of 93.8% (95% CI, 86.9%-100.0%), a specificity of 98.7% (95% CI, 96.0%-100.0%),and an AUC of 0.966 (95% CI, 0.927-0.998).”
    Quote in the paperSubstantial claim
  • Findingown data

    At the slide level, the leading algorithms performed better than the time-constrained pathologists in the simulation exercise.

    “On the slide level, the leading algorithms performed better than the pathologists WTC in the simulation exercise.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The simulated time-constrained exercise mimics routine diagnostic pathology workflow.
  • Findingown data

    The pathologist WOTC achieved a task 2 AUC of 0.966 (95% CI, 0.927-0.998).

    “Pathologist WOTC | 0.724 | 0.966 | | | | Expert pathologist who assessed”
    Quote in the paperSubstantial claim
  • Findingown data

    The time-constrained pathologists achieved a mean specificity of 98.5% (95% CI, 97.9%-99.1%).

    “They achieved a mean sensitivity of 62.8% (95% CI, 58.9%-71.9%)with a mean specificity of 98.5% (95% CI, 97.9%-99.1%).The mean AUC was 0.810 (range, 0.738-0.884)”
    Quote in the paperSubstantial claim
  • Findingown data

    The pathologist without time constraint produced no false-positive detections in task 1.

    “No false-positives were produced in task 1 (ie, nontumorous tissue indicated as metastasis) by the pathologist WOTC”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 16
  • The quote only partly states this claim. 3
  • May combine two claims. 11

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    All 32 submitted algorithms used a discriminative learning approach to identify metastases in whole-slide images.

    “All of the 32 algorithms submitted to CAMELYON16 used a discriminative learning approach to identify metastases in whole-slide images.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The mean AUC of the 11 pathologists WTC was 0.810 (95% CI, 0.750-0.869).

    “Mean pathologists WTC | | 0.810 | | | | The mean performance of 11”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Deep learning-based algorithms performed significantly better than other methods, and the 19 top-performing algorithms in both tasks all used deep convolutional neural networks.

    “Overall, deep learning-based algorithms performed significantly better than other methods: the 19 topperforming algorithms in both tasks all used deep convolutional neural networks as the underlying methodology (Table 2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The earlier HMS and MGH I submission achieved an overall AUC of 0.964 (95% CI, 0.928-0.989) and ranked third.

    “An earlier submission by this team, HMS and MGH I, achieved an overall AUC of 0.964 (95% CI, 0.928-0.989)and ranked third.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The pathologist without time constraint missed 27.6% of individual metastases, with a lesion-level true-positive fraction of 72.4% (95% CI, 64.3%-80.4%).

    “but 27.6% of individual metastases were not identified (lesion level, truepositive fraction, 72.4% [95% CI, 64.3%-80.4%])that manifested when IHC staining was performed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The pathologist WOTC produced no false positives and achieved a true-positive fraction of 0.724 for detecting and localizing metastatic regions.

    “The pathologist did not produce any false-positives and achieved a true-positive fraction of 0.724 for detecting and localizing metastatic regions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In a cross-sectional evaluation of 32 challenge algorithms, seven deep learning algorithms discriminated sentinel lymph node metastases better than a panel of 11 pathologists working under simulated time constraints, with the best algorithm AUC 0.994 and best pathologist AUC 0.884.

    “In cross-sectional analyses that evaluated 32 algorithms submitted as part of a challenge competition, 7 deep learning algorithms showed greater discrimination than a panel of 11 pathologists in a simulated time-constrained diagnostic setting, with an area under the curve of 0.994 (best algorithm) vs 0.884 (best pathologist).”
    • Also stated in: § Task 2: Whole-Slide Image Classification · plain
      “Overall, 7 of the 32 submitted algorithms had significantly higher AUCs than the pathologists WTC (see Table 2 for the individual P values calculated using MRMC ROC analysis).”
    • Also stated in: § Discussion · boosted
      “The CAMELYON16 challenge demonstrated that some deep learning algorithms were able to achieve a better AUC than a panel of 11 pathologists WTC participating in a simulation exercise for detection of lymph node metastases of breast cancer.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best algorithm achieved a true-positive fraction similar to the pathologist without time constraint when producing a mean of 1.25 false-positive lesions per 100 whole-slide images, and performed better when slightly more false-positive findings were allowed.

    “The best algorithm achieved similar true-positive fraction as the pathologist WOTC when producing a mean of 1.25 false-positive lesions in 100 whole-slide images and performed better when allowing for slightly more falsepositive findings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The image set was randomly divided into a training set of 270 and a test set of 129.

    “The set of images was randomly divided into a training (n = 270) and a test set (n = 129; details in Table 1).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationreasoning

    Auxiliary strategies to improve system generalization and performance appeared more important than the specific network architecture used.

    “system generalization and performance seemed more important.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Previous studies on diagnostic imaging tasks in which deep learning reached human-level performance, such as detection of diabetic retinopathy in retinal fundus photographs, used a reference standard based on the consensus of human experts.

    “Previous studies on diagnostic imaging tasks in which deep learning reached human-level performance, such as detection of diabetic retinopathy in retinal fundus photographs, used a reference standard based on the consensus of human experts.3”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The test data set was enriched with cases containing metastases, particularly micrometastases, and is therefore not directly comparable with the mix of cases encountered in clinical practice.

    “The test data set on which algorithms and pathologists were evaluated was enriched with cases containing metastases and, specifically, micrometastases and, thus, is not directly comparable with the mix of cases pathologists encounter in clinical practice.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most sentinel lymph nodes do not contain metastases.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Pathologists may rely on immunohistochemical staining and on the knowledge that negative hematoxylin and eosin slides will undergo additional immunohistochemical review.

    “pathologists may rely on IHC staining and the knowledge that all hematoxylin and eosin-slides with negative findings will undergo additional review with the use of IHC.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In most hospitals, pathologists request additional immunohistochemical staining in equivocal cases.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    Assessment without a time constraint does not give a fair measure of accuracy in routine diagnostic practice.

    “Assessment without time constraint does not yield a fair measure of the accuracy of the routine diagnostic process.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductnone given

    This study generated a reference standard using additional immunohistochemical staining, yielding an independent reference against which human pathologists could also be compared.

    “staining, yielding an independent reference against which human pathologists could also be compared.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Less time constraint on task completion may increase the accuracy of sentinel lymph node diagnostic review.

    “Less time constraint on task completion may increase the accuracy of SLN diagnostic review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Algorithm runtime was not recorded or included as a factor in the evaluation.

    “algorithm runtime was not recorded nor included as a factor in the evaluation, but it might influence the suitability for use in, for example, frozen section analysis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The simulation exercise had pathologists WTC review 129 hematoxylin and eosin-stained slides in about 2 hours to determine the presence of macroscopic or microscopic sentinel lymph node metastasis.

    “the simulation exercise invited pathologists WTC to review a series of 129 hematoxylin and eosinstained slides in about 2 hours to determine the presence of macroscopic or microscopic SLN metastasis.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The percentile bootstrap method was used to construct 95% CIs for FROC true-positive fraction scores and AUCs.

    “Thepercentilebootstrapmethod was used to construct 95% CIs for FROC true-positive fraction scores (FROC scores) and AUCs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The funders and sponsors had no role in the design and conduct of the study, data collection, management, analysis, interpretation, manuscript preparation, review, approval, or the decision to submit for publication.

    “The funders and sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The time-constrained pathologists assessed the same glass slides that were used to test the algorithms.

    “The pathologists WTC assessed the same set of glass slides used for testing the algorithms (which used digitized whole-slide images of these glass slides).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Immunohistochemistry is the most accurate method for metastasis evaluation and has little interpretation variability.

    “IHC is the most accurate method for metastasis evaluation and has little interpretation variability. [7][8]][9]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7][8]][9]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The algorithms were trained to discriminate normal from cancerous tissue within lymph node histological architecture and might be unable to identify rare co-occurring pathologies such as lymphoma, sarcoma, or infection.

    “algorithms were specifically trained to discriminate between normal and cancerous tissue in the background of lymph node histological architecture, but they might be unable to identify rare events such as co-occurring pathologies (eg, lymphoma, sarcoma, or infection).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Detection of other pathologies in the sentinel lymph node was not included in this study.

    “The detection of other pathologies in the SLN, which is relevant in routine diagnostics, was not included in this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    No power analysis was used to predetermine the sample size.

    “No prior data were available for the performance of algorithms in this task.Therefore, no power analysis was used to predetermine the sample size.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: No prior data were available for algorithm performance in this task.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    In this study, each pathologist was given a single hematoxylin and eosin-stained slide per patient to determine the presence or absence of breast cancer metastasis.

    “In this study, every pathologist was given 1 single hematoxylin and eosin-stained slide per patient to determine the presence or absence of breast cancer metastasis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    P values for the comparison of each algorithm with the pathologists WTC were adjusted for multiple comparisons using the Bonferroni correction, multiplying by the number of comparisons (32).

    “The P values were adjusted for multiple comparisons using the Bonferroni correction, in which the P values are multiplied by the number of comparisons (32; comparison of the 32 submitted algorithms with the panel of pathologists).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The 2-hour review time used in the simulation does not represent the work pace in other settings.

    “Although feasible in the context of this simulation, this does not represent the work pace in other settings.”
    Quote in the paperNot a substantial claim
28 of 28

  • The paper relies on this without support.

    The clinical value of isolated tumor cells in a sentinel lymph node is disputed.

    “Because the clinical value of having only isolated tumor cells in an SLN is disputed, we did not include such slides in our study”

  • The paper relies on this without support.

    The simulated time-constrained exercise mimics routine diagnostic pathology workflow.

    “To mimic routine diagnostic pathology workflow, we asked 11 pathologists to independently assess the 129 slides in the test set in”

  • The paper relies on this without support.

    The panel of 11 pathologists is representative of pathologists in routine practice.

    “The panel of the 11 pathologists (mean age, 47.7 years [range, 31-61]) included 1 resident pathologist (3-year resident) and 10 practicing pathologists”

  • The paper relies on this without support.

    Comparing algorithms on whole-slide images with pathologists on glass slides is valid despite the different modalities.

    “The panel of 11 pathologists with time constraint (pathologists WTC), however, did their assessment on the corresponding glass slides for those images because diagnosing is most commonly done using a microscope in pathology labs.”

  • The paper relies on this without support.

    In the Netherlands, use of immunohistochemistry is mandatory for cases with negative findings on H&E-stained slides.

    “The setting resembled diagnostic practice in the Netherlands, where use of IHC is mandatory for cases with negative findings on hematoxylin and eosin-stained slides.”

  • The paper relies on this without support.

    No prior data were available for algorithm performance in this task.

    “No prior data were available for the performance of algorithms in this task.Therefore, no power analysis was used to predetermine the sample size.”

  • The paper relies on this without support.

    Most sentinel lymph nodes do not contain metastases.

    “Given the reality that most SLNs do not contain metastases, the data set curation was needed to achieve a well-rounded representation of what is encountered in clinical practice without including an exorbitant number of slides.”

  • The paper relies on this without support.

    In real clinical settings, sections from multiple levels are evaluated for every lymph node.

    “In a real clinical setting, sections from multiple levels are evaluated for every lymph node.”

  • The paper relies on this without support.

    In most hospitals, pathologists request additional immunohistochemical staining in equivocal cases.

    “in most hospitals pathologists request additional IHC staining in equivocal cases.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CNN has achieved expert-level performances in various fields of medical research, including diabetic retinopathy screening, skin lesion classification, and lymph node metastasis detection.

    “Medical research is no exception, as CNN has achieved expert-level performances in various fields. Gulshan et al. [4], Esteva et al. [5], and Ehteshami Bejnordi et al. [6] demonstrated the potential of deep learning for diabetic retinopathy screening, skin lesion classification, and lymph node metastasis detection, respectively.”

    From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [6]

  • Certain deep learning algorithms surpassed a group of 11 pathologists in detecting lymph node metastases in breast cancer patients.

    “In a notable study, Bejnordi et al. (2017) found that certain deep learning algorithms surpassed a group of 11 pathologists in detecting lymph node metastases in breast cancer patients.”

    From A perspective on integrating digital pathology, proteomics, clinical data and AI analytics in cancer research (2025) · cites this paper as [24]

  • In CAMELYON, many models performed better than a reference panel of pathologists working under time constraints designed to mimic routine workflows.

    “This evaluation includes comparison with a reference panel of pathologists working on the same data set with predefined time constraints designed to mimic routine pathology workflows, where many of the models in the challenge exhibited higher performance than the panel of pathologists.”

    From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [45]

  • Deidentification and public release of data, often through consortium models, is an option for data sharing; CAMELYON and BraTS challenges have made large-scale deidentified datasets available, and the Nightingale Open Science Initiative aims to release more.

    “The second option is deidentification and public release of data, usually via a consortium model.”

    From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Figure caption reverses the pathologist groups' slide formats

    Minor

    The Figure 3 caption says the time-constrained pathologist assessed whole-slide images and the 11 pathologists without time constraints assessed glass slides. The methods state the opposite: the single pathologist without a time constraint assessed whole-slide images, while the panel of 11 with time constraints assessed glass slides.

1 of 1

Cites

Flagged first, then by citations

2 of the 40 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., Balkenhol, M., Geessink, O., Stathonikos, N., van Dijk, M. C., Bult, P., Beca, F., Beck, A. H., Wang, D., Khosla, A., Gargeya, R., Irshad, H., Zhong, A., Dou, Q., Li, Q., Chen, H., Lin, H.-J., Heng, P.-A., Haß, C., Bruni, E., Wong, Q., Halici, U., Öner, M. Ü., Cetin-Atalay, R., Berseth, M., Khvatkov, V., Vylegzhanin, A., Kraus, O., Shaban, M., Rajpoot, N., Awan, R., Sirinukunwattana, K., Qaiser, T., Tsang, Y.-W., Tellez, D., Annuscheit, J., Hufnagl, P., Valkonen, M., Kartasalo, K., Latonen, L., Ruusuvuori, P., Liimatainen, K., Albarqouni, S., Mungal, B., George, A., Demirci, S., Navab, N., Watanabe, S., Seno, S., Takenaka, Y., Matsuda, H., Ahmady Phoulady, H., Kovalev, V., Kalinovsky, A., Liauchuk, V., Bueno, G., Fernandez-Carrobles, M. M., Serrano, I., Deniz, O., Racoceanu, D., & Venâncio, R.
Affiliations
  • Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
  • Eindhoven University of Technology (NL)
  • Department of Pathology, University Medical Center Utrecht, Utrecht, the Netherlands
  • Radboud University Medical Center (NL)
  • Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
  • University Medical Center Utrecht (NL)
  • Rijnstate Hospital, Arnhem, the Netherlands
  • BeckLab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
  • Beth Israel Deaconess Medical Center (US)
  • PathAI (United States) (US)
  • Harker School, San Jose, California
  • Harvard University (US)
  • Chinese University of Hong Kong (HK)
  • Chinese University of Hong Kong, Hong Kong, China
  • ExB Research and Development GmbH, Munich, Germany
  • Munich Business School (DE)
  • Department of Electrical and Electronics Engineering, Middle East Technical University, Ankara, Turkey
  • Middle East Technical University (TR)
  • Cancer System Biology Laboratory, Graduate School of Informatics, Middle East Technical University, Ankara, Turkey
  • NLP LOGIX, Jacksonville, Florida
  • Smart Imaging Technologies, Houston, Texas
  • University of Toronto (CA)
  • University of Warwick (GB)
  • University Hospital Coventry (GB)
  • Qatar University (QA)
  • HTW Berlin - University of Applied Sciences (DE)
  • Tampere University of Technology (FI)
  • Tampere University (FI)
  • Technical University of Munich (DE)
  • The University of Osaka (JP)
  • University of South Florida, Tampa, Florida
  • National Academy of Sciences of Belarus (BY)
  • University of Castilla-La Mancha (ES)
  • Visilab, University of Castilla-La Mancha, Ciudad Real, Spain
  • Inserm (FR)
  • Sorbonne Université (FR)
ORCID
Venue
JAMA
Publisher
American Medical Association (AMA)
Published
2017-12-12
Volume
318
Issue
22
Pages
2199
ISSN
  • 0002-9955 (ISSN-L)
  • 0098-7484
DOI
10.1001/jama.2017.14585
PMC
PMC5820737
OpenAlex
W2772723798
Semantic Scholar
ba913e2c03ece1c75f0af4d16dd11c7ffbc6e3ba
Cited by
3,293
References
40
Citations by year
2026: 301 · 2025: 388 · 2024: 430 · 2023: 414 · 2022: 429 · 2021: 457 · 2020: 421 · 2019: 321 · 2018: 119 · 2017: 4 · 2015: 1 · 2014: 1
FWCI
141.10
Type
journal article
Language
en
Open access
Licence
other-oa
Locations
  • JAMA (publishedVersion, other-oa)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • OpenMETU (Middle East Technical University) (submittedVersion, cc-by-nc-nd)
  • TU/e Research Portal (submittedVersion)
  • Qatar University QSpace (Qatar University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
  • Utrecht University Repository (Utrecht University) (submittedVersion)
Funders
  • European Commission (601040)
  • Japan Society for the Promotion of Science (15K00403)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
lymph node metastasis, breast cancer, whole slide images, deep learning, computer-aided diagnosis, H&E staining, diagnostic accuracy, immunohistochemistry validation, breast cancer staging, lymph node staging, false positives, diagnostic confidence
MeSH terms
Machine Learning, Pathologists, Algorithms, Breast Neoplasms, Female, Humans, Lymphatic Metastasis, Pathology, Clinical, ROC Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review