CoChat
Sign Up
Cited by 315AI in cancer detectionMedical Image Segmentation TechniquesCell Image Analysis Techniques

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Structured crowdsourcing enables convolutional segmentation of histology images

Open access
Amgad, Mohamed, Elfandy, Habiba, Hussein, Hagar, Atteya, Lamees A, Elsebaie, Mai A T, Abo Elnasr, Lamia S, Sakr, Rokia A, Salem, Hazem S E, Ismail, Ahmed F, Saad, Anas M, Ahmed, Joumana, Elsebaie, Maha A T, Rahman, Mustafijur, Ruhban, Inas A, Elgazar, Nada M, Alagha, Yahya, Osman, Mohamed H, Alhusseiny, Ahmed M, Khalaf, Mariam M, Younes, Abo-Alela F, Abdulkarim, Ali, Younes, Duaa M, Gadallah, Ahmed M, Elkashash, Ahmad M, Fala, Salma Y, Zaki, Basma M, Beezley, Jonathan, Chittajallu, Deepak R, Manthey, David, Gutman, David A, Cooper, Lee A D·2019·Bioinformatics
DOI 10.1093/bioinformatics/btz083PMC PMC6748796
Open this paper in CoChat — free accountOpen PDF

Abstract

MOTIVATION: While deep-learning algorithms have demonstrated outstanding performance in semantic image segmentation tasks, large annotation datasets are needed to create accurate models. Annotation of histology images is challenging due to the effort and experience required to carefully delineate tissue structures, and difficulties related to sharing and markup of whole-slide images. RESULTS: We recruited 25 participants, ranging in experience from senior pathologists to medical students, to delineate tissue regions in 151 breast cancer slides using the Digital Slide Archive. Inter-participant discordance was systematically evaluated, revealing low discordance for tumor and stroma, and higher discordance for more subjectively defined or rare tissue classes. Feedback provided by senior participants enabled the generation and curation of 20 000+ annotated tissue regions. Fully convolutional networks trained using these annotations were highly accurate (mean AUC=0.945), and the scale of annotation data provided notable improvements in image classification accuracy. AVAILABILITY AND IMPLEMENTATION: Dataset is freely available at: https://goo.gl/cNM4EL. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bioinformatics/btz083

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Study overview. (A) Slides from the TNBC cohort were reviewed for difficulty and the study coordinator selected a single representative ROI in each slide. (B) Participants were recruited on social media from medical student interest groups. Documentation and instructional videos were developed to train participants in breast cancer pathology and the use of DSA annotation tools. A spreadsheet lists slide-level descriptions of histologic features for each of the 151 images to aid in training. (C) Participants were each assigned six slides based on experience. Challenging slides were assigned to faculty/pathology residents, while standard slides were distributed among all participants. (D) The DSA was used by participants to draw the outlines of tissue regions in their assigned slides/ROIs. A Slack workspace enabled less experienced users to ask questions and receive guidance from the more experienced users. (E) Ten evaluation ROIs were identified in the slides and were annotated by all participants in an unsupervised manner to enable inter-participant comparisons. (F) Agreement between each pair of participants was evaluated using the Dice coefficient to generate an inter-participant discordance matrix
Fig. 1. Study overview. (A) Slides from the TNBC cohort were reviewed for difficulty and the study coordinator selected a single representative ROI in each slide. (B) Participants were recruited on social media from medical student interest groups. Documentation and instructional videos were developed to train participants in breast cancer pathology and the use of DSA annotation tools. A spreadsheet lists slide-level descriptions of histologic features for each of the 151 images to aid in training. (C) Participants were each assigned six slides based on experience. Challenging slides were assigned to faculty/pathology residents, while standard slides were distributed among all participants. (D) The DSA was used by participants to draw the outlines of tissue regions in their assigned slides/ROIs. A Slack workspace enabled less experienced users to ask questions and receive guidance from the more experienced users. (E) Ten evaluation ROIs were identified in the slides and were annotated by all participants in an unsupervised manner to enable inter-participant comparisons. (F) Agreement between each pair of participants was evaluated using the Dice coefficient to generate an inter-participant discordance matrixCC BY · PubMed Central
Fig. 2. Screenshot of the DSA and HistomicsTK web interface. The main viewport allows panning and zooming within the slide. Annotations are grouped by class into layers (middle right panel) whose style properties like color and fill can be adjusted (bottom right panel). Other features include: controlling annotation transparency, an interactive mode to highlight individual annotations, and ability to download the WSI, regions of interest or annotations. Annotation properties can also be programmatically manipulated using the DSA API
Fig. 2. Screenshot of the DSA and HistomicsTK web interface. The main viewport allows panning and zooming within the slide. Annotations are grouped by class into layers (middle right panel) whose style properties like color and fill can be adjusted (bottom right panel). Other features include: controlling annotation transparency, an interactive mode to highlight individual annotations, and ability to download the WSI, regions of interest or annotations. Annotation properties can also be programmatically manipulated using the DSA APICC BY · PubMed Central
Fig. 3. Evaluation slide set concordance and model accuracy. (A) Inter-participant discordance matrices for SP, JP, NP and AL. (B) 2-D MDS plots of participant discordance. (C, D) Testing accuracy and confusion of comparison models trained on evaluation set ROIs from SPs (cyan) and NPs (magenta), measured against post-correction masks from the core set. Confusion matrix values are percentages relative to total pixel count. (Color version of this figure is available at Bioinformatics online.)
Fig. 3. Evaluation slide set concordance and model accuracy. (A) Inter-participant discordance matrices for SP, JP, NP and AL. (B) 2-D MDS plots of participant discordance. (C, D) Testing accuracy and confusion of comparison models trained on evaluation set ROIs from SPs (cyan) and NPs (magenta), measured against post-correction masks from the core set. Confusion matrix values are percentages relative to total pixel count. (Color version of this figure is available at Bioinformatics online.)CC BY · PubMed Central
Fig. 4. Model performance over the testing set. (A) Visualization of full semantic segmentation model predictions on testing set regions of interest. Color codes used: red (tumor); transparent (stroma); cyan (inflammatory infiltrates); yellow (necrosis). (B) Area under ROC curve for semantic segmentation algorithm, broken down by region class. (C) Effect of training sample size on scale-dependent patch classification models. Each point represents the macro-average AUC of a single model, trained on different sets of randomly selected slides. (Color version of this figure is available at Bioinformatics online.)
Fig. 4. Model performance over the testing set. (A) Visualization of full semantic segmentation model predictions on testing set regions of interest. Color codes used: red (tumor); transparent (stroma); cyan (inflammatory infiltrates); yellow (necrosis). (B) Area under ROC curve for semantic segmentation algorithm, broken down by region class. (C) Effect of training sample size on scale-dependent patch classification models. Each point represents the macro-average AUC of a single model, trained on different sets of randomly selected slides. (Color version of this figure is available at Bioinformatics online.)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 1

  1. Su, 2012, Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence

Cited by

Sorted by citations

See all 315

Related

Metadata

Authors
Amgad, M., Elfandy, H., Hussein, H., Atteya, L. A., Elsebaie, M. A. T., Abo Elnasr, L. S., Sakr, R. A., Salem, H. S. E., Ismail, A. F., Saad, A. M., Ahmed, J., Elsebaie, M. A. T., Rahman, M., Ruhban, I. A., Elgazar, N. M., Alagha, Y., Osman, M. H., Alhusseiny, A. M., Khalaf, M. M., Younes, A.-A. F., Abdulkarim, A., Younes, D. M., Gadallah, A. M., Elkashash, A. M., Fala, S. Y., Zaki, B. M., Beezley, J., Chittajallu, D. R., Manthey, D., Gutman, D. A., & Cooper, L. A. D.
Affiliations
  • Emory University (US)
  • Department of Pathology, National Cancer Institute , Cairo, Egypt
  • Cairo University (EG)
  • Ministry of Health and Population (EG)
  • Ain Shams University (EG)
  • Department of Medicine, Menoufia University , Menoufia, Egypt
  • Menoufia University (EG)
  • Alexandria University (EG)
  • University of Chittagong (BD)
  • Damascus University (SY)
  • Mansoura University (EG)
  • Zagazig University (EG)
  • Batterjee Medical College (SA)
  • Suez Canal University (EG)
  • Kitware (United States) (US)
Corresponding author
Cooper, Lee A D
ORCID
Venue
Bioinformatics
Publisher
Oxford University Press (OUP)
Published
2019-02-06
Volume
35
Issue
18
Pages
3461-3467
ISSN
  • 1367-4803 (ISSN-L)
  • 1367-4811
DOI
10.1093/bioinformatics/btz083
PMC
PMC6748796
OpenAlex
W2922239620
Cited by
315
References
11
Citations by year
2026: 41 · 2025: 73 · 2024: 61 · 2023: 60 · 2022: 42 · 2021: 20 · 2020: 17 · 2019: 1
FWCI
9.12
Type
journal article
Language
en
Open access
Licence
Locations
  • Bioinformatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Greater South Information System
Funders
  • U.S. National Institutes of Health
  • National Cancer Institute grants (U01CA220401, U24CA194362)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
Keywords
breast cancer, whole slide images, annotated dataset, fully convolutional network
MeSH terms
Algorithms, Breast Neoplasms, Histological Techniques, Humans, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review