AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Solving the recurrent issue of inadequate labeling during the analysis of medical images can be achieved by feeding the crowd-sourced input labels into the CNN.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
In the citing paper
“Albarquoni et al. [294] investigated the problem of insufficient labeling. They crowd-sourced the actual mitoses labeling in the histology images of breast cancer (from amateurs online).”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
BackgroundMethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
(2010) · International Conference on Machine Learning · Cited by 13,995
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
MethodMatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 1×
“vNet [34] and conducted on an Intel i7 machine with a GeForce”
MethodTowards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858 · cited 1×
“C ROWDSOURCING is a type of participative online activity in which an individual, an institution, a non-profit organization or a company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via flexible open call, the voluntary undertaking of a task [1].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Being inspired by prominent previous work in this field [18], [26], we define the specific contribution of our own work as: i) Learning of a multi-scale CNN model for mitosis detection, ii) Incorporation of aggregation schemes into CNN layers, and iii) Augmentation and retraining of the CNN model with crowd's annotati…”
Method
Research works printed but not linked: 1
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
Cited by
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
“Test (n=129 slides and images) RUMC 23 6 50 14 15 2 (1-14) 79”
BackgroundSelf-citationActivation Functions: Comparison of trends in Practice and Research for Deep Learning
Nwankpa, C. E., Ijomah, W., Gachagan, A., & Marshall, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03378 · Cited by 1,240
“to speech 4 recognition [27], [28], segmentation [29], [30], scene understanding and description [31], [32], machine translation [33], [34] test to speech systems [35], [36], cancer detection systems [37], [38], [39], finger print detection [40], [41], weather forecast [42], [43], self-driving cars [44], [45], and othe…”
Background
Metadata
- Authors
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N.
- Affiliations
- German Center for Neurodegenerative Diseases (DE)
- Technical University of Munich (DE)
- University of Oxford (GB)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1313-1321
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2339885376
- Cited by
- 606
- References
- 38
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 59 · 2023: 59 · 2022: 60 · 2021: 92 · 2020: 89 · 2019: 80 · 2018: 73 · 2017: 46 · 2016: 10
- FWCI
- 57.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- DZNE Pub (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- mitosis detection, breast cancer histopathology, crowdsourced annotation, noisy annotations, data aggregation, convolutional neural networks, CrowdFlower
- MeSH terms
- Machine Learning, Breast Neoplasms, Female, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Mitosis, Neural Networks, Computer, Video Games, Internet, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images. https://app.cochat.ai/papers/10.1109/tmi.2016.2528120
- The paper
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N. (2016). AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images. IEEE Transactions on Medical Imaging. https://doi.org/10.1109/tmi.2016.2528120
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.