AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Solving the recurrent issue of inadequate labeling during the analysis of medical images can be achieved by feeding the crowd-sourced input labels into the CNN.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
In the citing paper
“Albarquoni et al. [294] investigated the problem of insufficient labeling. They crowd-sourced the actual mitoses labeling in the histology images of breast cancer (from amateurs online).”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
BackgroundMethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
(2010) · International Conference on Machine Learning · Cited by 13,995
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
MethodMatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 1×
“vNet [34] and conducted on an Intel i7 machine with a GeForce”
MethodTowards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858 · cited 1×
“C ROWDSOURCING is a type of participative online activity in which an individual, an institution, a non-profit organization or a company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via flexible open call, the voluntary undertaking of a task [1].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Being inspired by prominent previous work in this field [18], [26], we define the specific contribution of our own work as: i) Learning of a multi-scale CNN model for mitosis detection, ii) Incorporation of aggregation schemes into CNN layers, and iii) Augmentation and retraining of the CNN model with crowd's annotati…”
Method
Research works printed but not linked: 1
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
Cited by
Items 301–400 of 613
Singh, A. P., Singh, A., Kumar, A., Agarwal, H., Yadav, S., & Gupta, M. (2024) · doi:10.3390/engproc2024062002 · Cited by 14
Machine learning from crowds: A systematic review of its applications
Rodrigo, E. G., Aledo, J. A., & Gámez, J. A. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1288 · Cited by 15
“…of pulmonary nodules (Kaya & Can, 2015); prematurity diagnosis in retinopathy (Ataer-Cansizoglu et al., 2015); mitosis detection in breast cancer (Albarqouni et al., 2016); melanoma cell segmentation (Sameki et al., 2016); sperm analysis (Chang et al., 2017); and segmentation of chromosomes…”
BackgroundGupta, R. K., Kaur, M., & Manhas, J. (2019) · Lecture notes in electrical engineering · doi:10.1007/978-3-030-29407-6_12 · Cited by 14
Deep-Learning-Based Polar-Body Detection for Automatic Cell Manipulation
Wang, Y., Liu, Y., Sun, M., & Zhao, X. J. (2019) · Micromachines · doi:10.3390/mi10020120 · Cited by 14
Artificial intelligence for pathology
Xing, F., Zhang, X., & Cornish, T. C. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-821259-2.00011-9 · Cited by 16
Deep YOLO-Based Detection of Breast Cancer Mitotic-Cells in Histopathological Images
Zorgani, M. M. A., Mehmood, I., & Ugail, H. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-3880-0_35 · Cited by 17
Kentley, J., Weber, J., Liopyris, K., Braun, R. P., Marghoob, A. A., Quigley, E. A., Nelson, K. C., Prentice, K., Duhaime, E., Halpern, A. C., et al. (2023) · JMIR Medical Informatics · doi:10.2196/38412 · Cited by 13
Optimization of deep learning based segmentation method
İnik, Ö., & Ülker, E. (2022) · Soft Computing · doi:10.1007/s00500-021-06711-3 · Cited by 13
“CNN models have been used in the identification of breast cancer (Albarqouni et al. 2016; Bayramoglu et al. 2016; Chen et al. 2016; Cires¸an et al. 2013; Gao et al. 2018; Rouhi et al. 2015; Spanhol et al. 2016; Wang et al. 2014a), segmentation of different tissues (Anthimopoulos et al. 2016;…”
MethodNoise-tolerant deep learning for histopathological image segmentation
Li, W., Qian, X., & Ji, J. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296848 · Cited by 15
Low Resource Sequence Tagging with Weak Labels
Simpson, E., Pfeiffer, J., & Gurevych, I. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6415 · Cited by 13
“Another approach by Albarqouni et al. (2016) integrates a CNN classifier for image classification into an aggrega- tion method based on expectation maximization (EM), while Yang et al. (2018) adapt a Bayesian neural network so that it can be trained concurrently with an annotator model, also using…”
MethodSun, Y., Xu, Z., Strell, C., Moro, C. F., Wärnberg, F., Dong, L., & Zhang, Q. (2018) · IEEE International Conference on Image Processing, Applications and Systems (IPAS) · doi:10.1109/ipas.2018.8708869 · Cited by 12
“of mitosis [11], [12], nuclei [13], [14] and lymphocytes [15].”
BackgroundLearning Effective Embeddings From Crowdsourced Labels: An Educational Case Study
Xu, G., Ding, W., Tang, J., Yang, S., Huang, G. Y., & Liu, Z. (2019) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00208 · Cited by 12
“introduced an additional crowdsourcing layer to embed the data aggregation process into convolutional neural network learning [23].”
Contextual Prior Constrained Deep Networks for Mitosis Detection With Point Annotations
Han, J., Wang, X., & Liu, W. (2021) · IEEE Access · doi:10.1109/access.2021.3079215 · Cited by 13
Pi, Y., Li, Q., Qi, X., Deng, D., & Zhang, Y. (2022) · Applied Intelligence · doi:10.1007/s10489-021-03140-5 · Cited by 13
“Deep neural networks, especially CNNs, which extract features in a data-driven manner, provide powerful support to alleviate the aforementioned drawbacks [2, 5].”
BackgroundApplication of Deep Learning System Technology in Identification of Women’s Breast Cancer
Fryan, L. H. A., Shomo, M. I., & Alazzam, M. B. (2023) · Medicina · doi:10.3390/medicina59030487 · Cited by 12
“In this article, we provide a novel idea for learning from crowds that directly handles data aggregation as a part of the convolutional neural network’s (CNN’s) learning process through an extra crowdsourcing layer (AggNet) [17].”
Exponential distance transform maps for cell localization
Li, B., Chen, J., Yi, H., Feng, M., Yang, Y., Zhu, Q., & Bu, H. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.107948 · Cited by 12
Traffic sign integrity analysis using deep learning
Acilo, J. N., Cruz, A. G. S. D., Kaw, M. K. L., Mabanta, M. D., Pineda, V. G. G., & Roxas, E. A. (2018) · IEEE International Colloquium on Signal Processing & Its Applications (CSPA) · doi:10.1109/cspa.2018.8368695 · Cited by 12
“In [16]-[17], its detection capability was used in mitosis and glaucoma.”
Liang, C., Bian, Z., Lv, W., Chen, S., Zeng, D., & Ma, J. (2018) · Physica Medica · doi:10.1016/j.ejmp.2018.10.016 · Cited by 12
Representation Learning from Limited Educational Data with Crowdsourced Labels
Wang, W., Xu, G., Ding, W., Huang, Y., Li, G., Tang, J., & Liu, Z. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3017122 · Cited by 16
“Albarqouni et al. introduced an additional crowdsourcing layer to embed the data aggregation process into convolutional neural network learning [61].”
MethodOutbreak COVID-19 in Medical Image Processing Using Deep Learning: A State-of-the-Art Review
Kaur, J., & Kaur, P. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09667-7 · Cited by 13
Dong, X., Li, M., Zhou, P., Deng, X., Li, S., Zhao, X., Wu, Y., Qin, J., & Guo, W. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01798-6 · Cited by 12
Image - based poultry disease detection using deep convolutional neural network
Mbelwa, H. (2021) · doi:10.58694/20.500.12479/1344 · Cited by 12
Learning from crowds for automated histopathological image segmentation
López-Pérez, M., Morales-Álvarez, P., Cooper, L., Felicelli, C., Goldstein, J., Vadasz, B., Molina, R., & Katsaggelos, A. K. (2024) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2024.102327 · Cited by 12
Federated Learning for Exploiting Annotators’ Disagreements in Natural Language Processing
Rodríguez-Barroso, N., Martínez‐Cámara, E., Camacho-Collados, J., Luzón, M. V., & Herrera, F. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00664 · Cited by 11
“…specific to text classification and can be applied to other NLP tasks, or even other machine learning related applications where we can find similar disagreement issues (Albarqouni et al., 2016; Beyer et al., 2020; Cabitza et al., 2019, 2020), but we leave this extended analysis for future work.”
BackgroundAfzaal, U., Su, Z., Sajjad, U., Stack, T., Lü, H., Niu, S., Akbar, A., Gürcan, M. N., & Niazi, M. K. K. (2025) · Patterns · doi:10.1016/j.patter.2025.101284 · Cited by 15
“…detection, nuclear segmentation, and region identification, enhancing nuclear morphometrics and other analyses, 11–13 cancer identification, 14–16 subtyping 17 and grading, 18–20 mitosis 21 and metastasis detection, 22,23 and survival 24–29 and treatment response prediction, 30,31 among many more.”
On the Synergy of Optimizers and Activation Functions: A CNN Benchmarking Study
Sayın, K. A., Gürsoy, N. K., Yolcu, T., & Gürsoy, A. (2025) · Mathematics · doi:10.3390/math13132088 · Cited by 11
Artificial intelligence in gamification: A systematic review
Bulut, A., & Ulusoy, F. (2026) · Entertainment Computing · doi:10.1016/j.entcom.2026.101112 · Cited by 12
Yu, X., Chen, H., Liang, M., Xu, Q. Y., & He, L. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09977-1 · Cited by 14
Learning from multiple inconsistent and dependent annotators to support classification tasks
Gil-González, J., Orozco, A. A., & Álvarez-Meza, A. M. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.045 · Cited by 11
Learning from biased crowdsourced labeling with deep clustering
Wu, M., Li, Q., Yang, F., Zhang, J., Sheng, V. S., & Hou, J. Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118608 · Cited by 12
A computational classification method of breast cancer images using the VGGNet model
Khan, A., Khan, A., Ullah, M., Alam, M. M., Bangash, J. I., & Su’ud, M. M. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.1001803 · Cited by 11
Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
Triana-Martinez, J. C., Gil-González, J., Fernandez-Gallego, J. A., Álvarez-Meza, A. M., & Castellanos-Dominguez, G. (2023) · Sensors · doi:10.3390/s23073518 · Cited by 11
“Likewise, some studies have adapted these ideas for DL techniques by incorporating an additional layer that contemplates multiple labelers [27,28].”
BackgroundMethodInfluentialDifference-Deformable Convolution With Pseudo Scale Instance Map for Cell Localization
Zhang, C., Chen, J., Li, B., Feng, M., Yang, Y., Zhu, Q., & Bu, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3329542 · Cited by 14
“Detection-based cell localization methods typically generate multiple candidate regions in the image and then classify and filter each region to identify those that truly contain cells [10], [11].”
BackgroundNing, Z., Zhang, X., Tu, C., Feng, Q., & Zhang, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2946478 · Cited by 11
A self-supervised feature-standardization-block for cross-domain lung disease classification
Li, X., Shen, L., Lai, Z., Li, Z., Yu, J., Pu, Z., Mou, L., Cao, M., Kong, H., Li, Y., et al. (2021) · Methods · doi:10.1016/j.ymeth.2021.05.007 · Cited by 11
Leveraging Convolutional Neural Network for COVID-19 Disease Detection Using CT Scan Images
Masud, M., Alshehri, M. D., Alroobaea, R., & Shorfuzzaman, M. (2021) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2021.016800 · Cited by 11
“CNN models are the most used techniques in several types of medical image classification for their superior extraction of various level features from images [14,15].”
MethodRegularized Chained Deep Neural Network Classifier for Multiple Annotators
Gil-González, J., Valencia-Duque, A. F., Álvarez-Meza, A. M., Orozco, A. A., & García-Moreno, A. (2021) · Applied Sciences · doi:10.3390/app11125409 · Cited by 14
Improve Learning from Crowds via Generative Augmentation
Chu, Z., & Wang, H. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467409 · Cited by 11
“Albarqouni et al. [1] extended this solution by replacing the logistic classifier with a deep neural network classifier.”
Deep Learning for Histopathological Image Analysis
Wemmert, C., Weber, J., Feuerhake, F., & Forestier, G. (2021) · Springer eBooks · doi:10.1007/978-3-030-71676-9_7 · Cited by 12
A review of big data analytics in the biomedical field
Jatmiko, W., Arsa, D. M. S., Wisesa, H. A., Jati, G., & Ma’Sum, M. A. (2016) · International Workshop on Big Data and Information Security (IWBIS) · doi:10.1109/iwbis.2016.7872886 · Cited by 11
“If our data is high dimensional, so we should face the curse of dimensionality called Hughes phenomenon [23].”
Rodrigues, F., & Pereira, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11506 · Cited by 10
Monaghan, J., Backholer, K., McKelvey, A., Christidis, R., Borda, A., Calyx, C., Crocetti, A., Driessen, C., & Zorbas, C. (2023) · Obesity Reviews · doi:10.1111/obr.13618 · Cited by 10
The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
López-Pérez, M., Morquecho, A., Schmidt, A., Pérez-Bueno, F., Castro, A. M., Mateos, J., & Molina, R. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108472 · Cited by 10
Zhu, L., Zhang, C., Dong, L., Rong, M., Gong, J. Y., & Meng, F. (2023) · Physica Scripta · doi:10.1088/1402-4896/acf007 · Cited by 10
A novel dilated contextual attention module for breast cancer mitosis cell detection
Li, Z., Li, X., Wu, W., Lyu, H., Tang, X., Zhou, C., Xu, F., Luo, B., Jiang, Y., Liu, X., et al. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1337554 · Cited by 10
Opinions Vary? Diagnosis First!
Wu, J., Fang, H., Yang, D., Wang, Z., Zhou, W., Shang, F., Yang, Y., & Xu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16434-7_58 · Cited by 10
Dif, N., & Elberrichi, Z. E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-50402-1_17 · Cited by 11
A crowdsourcing semi-automatic image segmentation platform for cell biology
Bafti, S. M., Ang, C. S., Hossain, M. M., Marcelli, G., Alemany-Fornés, M., & Tsaousis, A. D. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104204 · Cited by 10
Gandomkar, Z., Brennan, P., & Mello-Thoms, C. R. (2018) · International Workshop on Breast Imaging (IWBI) · doi:10.1117/12.2318320 · Cited by 11
“Deep learning has been widely used for detection of mitotic figures [18] and nuclei [19] within breast whole-slides.”
MethodDeep Learning for Biomedical Data Analysis: Techniques, Approaches, and Applications
(2021) · Springer eBooks · doi:10.1007/978-3-030-71676-9 · Cited by 12
Sumitha, A., & Isaac, R. S. R. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150428 · Cited by 9
“Histologically probing the microscopic realm, they unravel the intricate distinctions between normal tissue, benign formations, and the malignant lesions that herald a potential storm [2].”
BackgroundCMed: Crowd Analytics for Medical Imaging Data
Park, J. H., Nadeem, S., Boorboor, S., Marino, J., & Kaufman, A. E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2953026 · Cited by 11
“shown [39], and thus the output of our framework could be used for training data.”
MethodBreast Cancer Detection in Saudi Arabian Women Using Hybrid Machine Learning on Mammographic Images
Almalki, Y. E., Shaf, A., Ali, T. M., Aamir, M., Alduraibi, S. K., Almutiri, S. M., Irfan, M., Basha, M. A. A., Alduraibi, A. K., Alamri, A. M., et al. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.027111 · Cited by 9
“Nonlinear processing layers are introduced in deep learning with machine learning and artificial intelligence combined to extract information from the dataset [15].”
Backgrounddel Amor, R., Pérez-Cano, J., López-Pérez, M., Terrádez-Mas, L., Aneiros‐Fernández, J., Morales, S., Mateos, J., Molina, R., & Naranjo, V. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102686 · Cited by 13
A Little Truth Injection But a Big Reward: Label Aggregation With Graph Neural Networks
Ying, Z., Zhang, J., Li, Q., Wu, M., & Sheng, V. S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3338216 · Cited by 9
“AggNet [1] directly adds the aggregation into the learning process of the convolution neural networks (CNNs).”
BackgroundAugmenting Chest X-ray Datasets with Non-Expert Annotations
Cheplygina, V., Damgaard, C., Eriksen, T. N., Juodelyte, D., & Jiménez-Sánchez, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98688-8_10 · Cited by 11
“Each annotator first independently labeled the presence or absence of a tube (chest drain for NIH-CXR14, four types of tubes for PadChest).”
Deep assisted dense model based classification of invasive ductal breast histology images
Vidyarthi, A., & Patel, A. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-05947-2 · Cited by 9
Lin, S., Tran, C., Bandari, E., Romagnoli, T., Li, Y., Chu, M., Amirthakatesan, A. S., Dallmann, A., Kostiukov, A. O., Panizo, Á., et al. (2024) · International Journal of Surgical Pathology · doi:10.1177/10668969241234321 · Cited by 9
Data Engineering for Machine Learning in Women's Imaging and Beyond
Chen, C., Chou, S.-H. S., Brattain, L. J., Lehman, C. D., & Samir, A. E. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20464 · Cited by 10
“Unfortunately, efforts to create large labeled databases using multisite data or labeling sources introduce data standardization challenges [31, 40].”
BackgroundHager, G. D., Maier‐Hein, L., & Vedula, S. S. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-816176-0.00043-0 · Cited by 9
Aggregating Crowd Wisdom with Side Information via a Clustering-based Label-aware Autoencoder
Yin, L., Liu, Y., Zhang, W., & Yu, Y. H. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/214 · Cited by 9
“Recent work utilizes deep neural networks to enhance the learning capacity of the classifier [Albarqouni et al. , 2016; Dizaji and Huang, 2018; Rodrigues and Pereira, 2018].”
BackgroundMethodImage Segmentation of the Pulmonary Acinus Imaged by Synchrotron X-Ray Tomography
Arsić, B., Obrenović, M., Anić, M., Tsuda, A., & Filipović, N. (2019) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe.2019.00101 · Cited by 10
“In the last years, the paradigm has been applied extensively in several medical imaging applications, such as brain [30], [31], lung [32], [33], and breast [34], [35] imaging.”
BackgroundDizaj, S. B., & Valizadeh, P. (2021) · Research Square · doi:10.21203/rs.3.rs-952669/v1 · Cited by 10
Weighted Adversarial Learning From Crowds
Chen, Z., Jiang, L., Zhang, W., & Li, C. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3404353 · Cited by 8
“Despite the various methods proposed to address the challenges associated with LFC [16], [17], [18], [19], most existing LFC methods assume that all instances are benign [20], [21], [22].”
Playsourcing: A Novel Concept for Knowledge Creation in Biomedical Research
Albarqouni, S., Matl, S., Baust, M., Navab, N., & Demirci, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46976-8_28 · Cited by 12
Self-citationThe usage of deep learning algorithm in medical diagnostic of breast cancer
Parikesit, A. A., & Ramanto, K. N. (2019) · Malaysian Journal of Fundamental and Applied Sciences · doi:10.11113/mjfas.v15n2.1231 · Cited by 9
“Research that was conducted by Albarqouni et al. (2016) used histopathological images from at least 23 breast cancer patients.”
MethodSignet Ring Cells Detection in Histology Images with Similarity Learning
Sun, Y., Huang, X., Molina, E. G. L., Dong, L., & Zhang, Q. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098547 · Cited by 8
“in automatic mitosis detection in histology image of breast cancer [4, 9, 10].”
BackgroundShah, B., Alsadoon, A., Prasad, P. W. C., Al‐Naymat, G., & Beg, A. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10769-4 · Cited by 9
Learning from crowds with sparse and imbalanced annotations
Shi, Y., Li, S.-Y., & Huang, S.-J. (2022) · Machine Learning · doi:10.1007/s10994-022-06185-w · Cited by 9
“2008), medical diagnosis (Raykar et al., 2010; Shadi et al., 2016) and vision tagging (Filipe and Francisco, 2018; Welinder et al.”
MethodWeb Page Classification Algorithm Based on Deep Learning
Yu, Y. (2022) · Computational Intelligence and Neuroscience · doi:10.1155/2022/9534918 · Cited by 8
Learning from Crowds with Annotation Reliability
Cao, Z. W., Chen, E., Huang, Y., Shen, S., & Huang, Z. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592007 · Cited by 19
“In recent years, the need of labeled data has been increasing rapidly to apply supervised learning methods for practical problems [1, 11].”
BackgroundMethodLearning from Noisy Crowd Labels with Logics
Chen, Z., Sun, H., He, H., & Chen, P. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00011 · Cited by 10
“Our framework builds on a canonical probabilistic graphical model [12], [15] as a principled cornerstone.”
BackgroundMethodPeng, Y., Yi, X., Zhang, D., Zhang, L., Tian, Y., & Zhou, Z. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108559 · Cited by 8
Lakshmipriya, B., Jayanthi, K., Pottakkat, B., & Ramkumar, G. (2021) · Wiley eBooks · doi:10.1002/9781119785750.ch3 · Cited by 8
Defect Detection and Classification Algorithm of Metal Nanomaterials Based on Deep Learning
Xue, B., & Wu, Z. (2022) · Integrated ferroelectrics · doi:10.1080/10584587.2022.2065568 · Cited by 8
“In this way, the automatic learning of features can be realized without human participation [7,8].”
BackgroundGopakumar, G., & Subrahmanyam, G. R. K. S. (2019) · Smart innovation, systems and technologies · doi:10.1007/978-3-030-11479-4_11 · Cited by 8
Applications of Deep Learning Models in Laparoscopy for Gynecology
Gkrozou, F., Bais, V., Skentou, C., Kalaitzopoulos, D. R., Grigoriadis, G., Vatopoulou, A., Paschopoulos, M. E., & Daniilidis, A. (2025) · Medicina · doi:10.3390/medicina61081460 · Cited by 7
“…in multiple other medical domains, including polyp detection during colonoscopy [9], mitosis identification in breast cancer histology images [10], skin lesions classification in dermatology [11], brain tumor classification [12], and electrocardiogram (ECG) analysis [13], among numerous…”
Learning with less: A survey of deep learning in medical imaging under varying supervision levels
Kumari, S., & Singh, P. P. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103375 · Cited by 8
Learning From Crowds Using Graph Neural Networks With Attention Mechanism
Zhang, J., Wu, M., Sun, Z., & Zhou, C. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3378100 · Cited by 8
Wang, H. S., & Liang, W. (2024) · The American Journal of Surgical Pathology · doi:10.1097/pas.0000000000002223 · Cited by 7
Crowdsourced Emphysema Assessment
Ørting, S. N., Cheplygina, V., Petersen, J., Thomsen, L. H., Wille, M. M. W., & de Bruijne, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67534-3_14 · Cited by 7
“Due to the success of crowdsourcing in computer vision, there have also been several efforts to apply it to medical imaging [6, 2, 8, 1].”
BackgroundDey, M., Rana, S. P., Loretoni, R., Duranti, M., Sani, L., Vispa, A., Raspa, G., Ghavami, M., Dudley, S., & Tiberi, G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0271377 · Cited by 7
Deep Cellular Automata-Based Feature Extraction for Classification of the Breast Cancer Image
Tangsakul, S., & Wongthanavasu, S. (2023) · Applied Sciences · doi:10.3390/app13106081 · Cited by 7
Maulana, F. I., Adi, P. D. P., Lestari, D., Ariyanto, S. V., & Purnomo, A. (2023) · International Seminar on Intelligent Technology and Its Applications · doi:10.1109/isitia59021.2023.10221162 · Cited by 7
Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
Wu, J., Zhang, Y., Fang, H., Duan, L., Tan, M., Yang, W., Wang, C., Liu, H., Jin, Y., & Xu, Y. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3394045 · Cited by 10
“Another branch of study proposed to jointly estimate the multi-rater expertness and the classification (or segmentation) to handle classification (or segmentation) uncertainty in their tasks [8], [9], [10], [11], [12].”
BackgroundMethodInfluentialSharma, G., & Jindal, N. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12859-3 · Cited by 7
“There are extensive lesion annotations on every image [5, 8, 114].”
BackgroundRetractedMedical Monkeys: A Crowdsourcing Approach to Medical Big Data
Servadei, L., Schmidt, R., Eidelloth, C., & Maier, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-73805-5_9 · Cited by 9
“In specific tasks as cell mitosis in breast-cancer, Convolutional Neural Networks based on ground truth data generated by a crowd of non-experts has reached outstanding performance on diseased cells mitosis detection [25].”
BackgroundA Crowdsourcing Repeated Annotations System for Visual Object Detection
Hu, Y., Ou, Z., Xu, X., & Song, M. (2019) · International Conference on Vision, Image and Signal Processing (ICVISP) · doi:10.1145/3387168.3387242 · Cited by 8
“Such “crowdsourcing” is increasingly popular and has been used to annotate large datasets in, for example, image classification [1], medical image segmentation [10] and natural language processing [14].”
MethodImproved SegMitos framework for mitosis detection in breast cancer histopathology images
Sebai, M. (2020) · International Conference on Artificial Intelligence and Information Systems (ICAIIS) · doi:10.1109/icaiis49377.2020.9194877 · Cited by 7
“Recall F-score PANASONIC [15] 0.336 0.310 0.322 ISIK [15] 0.306 0.351 0.327 SURREY [15] 0.357 0.332 0.344 DTU [15] 0.427 0.555 0.483 IDSIA [15] 0.610 0.612 0.611 CUHK 0.690 0.310 0.427 AggNet [18] 0.441 0.424 0.433 SegMitos [4] 0.668 0.677 0.672 Our method 0.653 0.742 0.695”
InfluentialBayesian Detector Combination for Object Detection with Crowdsourced Annotations
Tan, Z. Q., Isupova, O., Carneiro, G., Zhu, X., & Li, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73036-8_19 · Cited by 8
Kashyap, S., Shukla, A. K., Naim, I., & Pal, S. (2025) · Biomedical Materials & Devices · doi:10.1007/s44174-025-00335-z · Cited by 6
“A sensitivity of 88.7% was achieved [33].”
InfluentialIntroduction to Digital Pathology from Historical Perspectives to Emerging Pathomics
Gupta, R., Kurç, T., & Saltz, J. H. (2021) · Springer eBooks · doi:10.1007/978-3-030-83332-9_1 · Cited by 7
Hybrid Deep Convolutional Neural Network Approach for Detecting Breast Cancer in Mammography Images
Nemade, V., Pathak, S. K., & Dubey, A. K. (2023) · International Journal of Electrical and Electronics Engineering · doi:10.14445/23488379/ijeee-v10i5p110 · Cited by 6
Generalized cross-entropy for learning from crowds based on correlated chained Gaussian processes
Gil-González, J., Daza-Santacoloma, G., Peña, D. A. C., Orozco, A. A., & Álvarez-Meza, A. M. (2025) · Results in Engineering · doi:10.1016/j.rineng.2024.103863 · Cited by 6
Active methodologies in health. Scientific production on gamification in health sciences
Robles, A. S., & González, M. E. P. (2019) · Science for Education Today · doi:10.15293/2658-6762.1903.13 · Cited by 6
Classification of Pneumonia Using Deep Convolutional Neural Network
Ahmed, A. M., Babikir, G. A., & Osman, S. M. (2022) · American Journal of Computer Science and Technology · doi:10.11648/j.ajcst.20220502.11 · Cited by 6
“Convolutional Neural Networks (CNNs) are the most favorite and popular deep learning models with superior achievements in the medical imaging domain since it provides high accuracy and impressive results compared with other models [8-11].”
ResultStain Colour Normalisation to Improve Mitosis Detection on Breast Histology Images
Hamidinekoo, A., & Reyer Zwiggelaar, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_25 · Cited by 6
Motepe, S., Hasan, A. N., Twala, B., Stopforth, R., & Alajarmeh, N. S. (2019) · International Aegean Conference on Electrical Machines and Power Electronics & International Conference on Optimization of Electrical and Electronic Equipment (ACEMP-OPTIM) · doi:10.1109/acemp-optim44294.2019.9007218 · Cited by 6
“Since AI’s inception in the mid-1900s, different AI techniques have been developed and applied in various fields to simplify humans’ life [4] [5] [6] [7].”
BackgroundBerthomier, T., Williams, D. P., d'Ales, B., & Dugelay, S. (2020) · Global Oceans 2020: Singapore – U.S. Gulf Coast · doi:10.1109/ieeeconf38699.2020.9389138 · Cited by 6
“But even in areas that typically suffer from limited data, CNNs have recently achieved state-of-the-art performance for classification tasks [2]–[5].”
BackgroundNINEPINS: Nuclei Instance Segmentation with Point AnnotationsPreprint
Yen, T.-A., Hsu, H. C., Pati, P., Gabrani, M., Foncubierta–Rodríguez, A., & Chung, P. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13556 · Cited by 6
“dels outperform traditional methods in the experiments. Deep learning is also widely used for medical image processing tasks, e.g. cell and membrane segmentation [13], breast cancer mitosis detection [1], and so forth. However, supervised training of deep learning models requires a large amount of annotations. This ca…”
Method
Metadata
- Authors
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N.
- Affiliations
- German Center for Neurodegenerative Diseases (DE)
- Technical University of Munich (DE)
- University of Oxford (GB)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1313-1321
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2339885376
- Cited by
- 606
- References
- 38
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 59 · 2023: 59 · 2022: 60 · 2021: 92 · 2020: 89 · 2019: 80 · 2018: 73 · 2017: 46 · 2016: 10
- FWCI
- 57.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- DZNE Pub (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- mitosis detection, breast cancer histopathology, crowdsourced annotation, noisy annotations, data aggregation, convolutional neural networks, CrowdFlower
- MeSH terms
- Machine Learning, Breast Neoplasms, Female, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Mitosis, Neural Networks, Computer, Video Games, Internet, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.