AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Solving the recurrent issue of inadequate labeling during the analysis of medical images can be achieved by feeding the crowd-sourced input labels into the CNN.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
In the citing paper
“Albarquoni et al. [294] investigated the problem of insufficient labeling. They crowd-sourced the actual mitoses labeling in the histology images of breast cancer (from amateurs online).”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
BackgroundMethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
(2010) · International Conference on Machine Learning · Cited by 13,995
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
MethodMatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 1×
“vNet [34] and conducted on an Intel i7 machine with a GeForce”
MethodTowards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858 · cited 1×
“C ROWDSOURCING is a type of participative online activity in which an individual, an institution, a non-profit organization or a company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via flexible open call, the voluntary undertaking of a task [1].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Being inspired by prominent previous work in this field [18], [26], we define the specific contribution of our own work as: i) Learning of a multi-scale CNN model for mitosis detection, ii) Incorporation of aggregation schemes into CNN layers, and iii) Augmentation and retraining of the CNN model with crowd's annotati…”
Method
Research works printed but not linked: 1
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
Cited by
Items 201–300 of 613
Histopathologic Image Processing: A ReviewPreprint
de Matos, J., Britto, A. D. S., Oliveira, L. S., & Koerich, A. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07900 · Cited by 40
“Mitosis is the matter of the works presented by Albarqouni et al. [131] and Zerhouni et al. [132].”
BackgroundXue, Y., Bigras, G., Hugh, J. C., & Ray, N. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2907093 · Cited by 34
“[20] used expectation maximization within a deep learning framework in an end-to-end fashion for mitosis detection.”
MethodStructured Probabilistic End-to-End Learning from Crowds
Chen, Z., Wang, H., Sun, H., Chen, P., Han, T., Liu, X., & Yang, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/210 · Cited by 32
“…to as the learning-from-crowds approach [Raykar et al. , 2010; Kajino et al. , 2012; Yan et al. , 2012; Rodrigues et al. , 2013; Bi et al. , 2014; Albarqouni et al. , 2016; Rodrigues and Pereira, 2018], simultaneously infers the true labels while training the classifier, allowing the two…”
BackgroundNeural Named Entity Recognition Using a Self-Attention Mechanism
Žukov-Gregorič, A., Bachrach, Y., Minkovsky, P., Coope, S., & Maksak, B. (2017) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2017.00104 · Cited by 33
“Deep neural networks have recently shown very strong performance in many tasks such as image classification and labeling [7], [8], [9], speech processing [10], [11], aggregating crowdsourced opinions [11], [12], and a wide range of natural language processing tasks and language modeling [13], [10], [14], [15], [16], […”
BackgroundChest disease radiography in twofold: using convolutional neural networks and transfer learning
Choudhary, P., & Hazra, A. (2019) · Evolving Systems · doi:10.1007/s12530-019-09316-2 · Cited by 32
Learning from crowds in digital pathology using scalable variational Gaussian processes
López-Pérez, M., Amgad, M. T., Morales-Álvarez, P., Ruiz, P., Cooper, L., Molina, R., & Katsaggelos, A. K. (2021) · Scientific Reports · doi:10.1038/s41598-021-90821-3 · Cited by 31
Sethy, P. K., & Behera, S. K. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11756-5 · Cited by 32
Deep Learning for Detection of Intracranial Aneurysms From Computed Tomography Angiography Images
Liu, X., Mao, J., Sun, N., Yu, X., Chai, L., Tian, Y., Wang, J., Liang, J., Tao, H., Yuan, L., et al. (2020) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00698-5 · Cited by 34
Early Experiences with Crowdsourcing Airway Annotations in Chest CT
Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H. A. W. M., & de Bruijne, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46976-8_22 · Cited by 31
Breast cancer diagnosis based on joint variable selection and Constructive Deep Neural Network
Zemouri, R. M., Omri, N., Devalland, C., Arnould, L., Chebel‐Morello, B., Noureddine Zerhouni, A., & Fnaiech, F. (2018) · Middle East Conference on Biomedical Engineering · doi:10.1109/mecbme.2018.8402426 · Cited by 33
“For the BC-CAD based on image processing, Deep Convolutional Neural Network (DCNN) is the mostly used neural network architecture in recent publications [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31].”
MethodWang, L., Chen, A., Zhang, Y., Wang, X., Zhang, Y., Shen, Q., & Xue, Y. (2020) · Diagnostics · doi:10.3390/diagnostics10040217 · Cited by 30
Li, J., Dou, Q., Yang, H., Liu, J., Fu, L., Zhang, Y., Zheng, L., & Zhang, D. (2021) · Tissue and Cell · doi:10.1016/j.tice.2021.101677 · Cited by 30
Trustable Co-Label Learning From Multiple Noisy Annotators
Li, S., Liu, T., Tan, J., Zeng, D., & Ge, S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3137752 · Cited by 40
“Many works in this direction propose an Expectation Maximization (EM) algo-rithm for jointly learning the levels of expertise of different annotators and the parameters of a classifier [22], [24], [25], [28].”
Deep Learning From Multiple Noisy Annotators as A Union
Wei, H., Xie, R., Feng, L., Han, B., & An, B. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3168696 · Cited by 52
“This prominent work inspired a number of follow-up works, such as Gaussian process classifiers [12], supervised latent Dirichlet allocation [13], and convolutional neural networks with softmax outputs [2], [14].”
BackgroundMethodLiu, A.-A., Lu, Y., Chen, M., & Su, Y. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2721438 · Cited by 30
“Deep learning methods have shown strong performance for detecting mitosis from single images [38], [39], [40].”
BackgroundAutomatic Lesion Detection in Periapical X-rays
Sajad, M., Shafi, I., & Ahmad, J. (2019) · International Conference on Electrical, Communication, and Computer Engineering (ICECCE) · doi:10.1109/icecce47252.2019.8940661 · Cited by 30
“Deep learning and the concept of transfer learning is recently gaining importance in the area of medical image processing [9]-[17], but no literature was found except Shreyansh et al [28] wherein used deep learning method for the classification of teeth lesions.”
MethodSu, Y., Lu, Y., Liu, J., Chen, M., & Liu, A.-A. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3052854 · Cited by 29
Statistical Methods for Annotation Analysis
Paun, S., Artstein, R., & Poesio, M. (2022) · Synthesis lectures on human language technologies · doi:10.2200/s01131ed1v01y202109hlt054 · Cited by 43
López-Pérez, M., Morales-Álvarez, P., Cooper, L., Molina, R., & Katsaggelos, A. K. (2023) · IEEE Access · doi:10.1109/access.2023.3237990 · Cited by 29
A region-based convolutional network for nuclei detection and segmentation in microscopy images
Liang, H., Cheng, Z., Zhong, H., Qu, A., & Chen, L. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103276 · Cited by 32
Classification of Breast Microscopic Imaging using Hybrid CLAHE-CNN Deep Architecture
Vidyarthi, A., Shad, J., Sharma, S., & Agarwal, P. (2019) · International Conference on Contemporary Computing · doi:10.1109/ic3.2019.8844937 · Cited by 28
“An approach that handles data aggregation directly as a part of the learning process of CNN through crowd sourcing layer is discussed in [16].”
BackgroundFast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment ClassificationPreprint
Sinha, V. B., Rao, S., & Balasubramanian, V. N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02781 · Cited by 33
“With the ever-growing need for large labeled datasets and the prohibitive costs of seeking experts to label large datasets, crowdsourcing has been used as a viable option for a variety of tasks, including sentiment scoring [3], opinion mining [39], general text processing [35], taxonomy creation [2], or domain-specifi…”
BackgroundScratch that! An Evolution-based Adversarial Attack against Neural NetworksPreprint
Jere, M., Rossi, L., Hitaj, B., Ciocarlie, G. F., Boracchi, G., & Koushanfar, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02316 · Cited by 29
“…[23], [28], [51], [53], machine translation [13], [14], [41] and reinforcement learning tasks [5], [49], [50], [57], and are being applied to a wide variety of domains, such as cancer detection from medical scans [1], [48], malware detection [30], [42], [56], speech recognition [20], and more.”
BackgroundUsing deep learning to enhance head and neck cancer diagnosis and classification
Gupta, P., & Malhi, A. K. (2018) · International Conference on System, Computation, Automation and Networking (ICSCAN) · doi:10.1109/icscan.2018.8541142 · Cited by 34
“[11] for detection of mitosis in breast cancer from histology images.”
BackgroundCReg-KD: Model refinement via confidence regularized knowledge distillation for brain imaging
Yang, Y., Guo, X., Ye, C., Xiang, Y., & Ma, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102916 · Cited by 29
“…that deep learning approaches have become state-of-the-art solutions to a variety of medical image analysis problems, such as lesion detection (Albarqouni et al., 2016; Bejnordi et al., 2017; Dou et al., 2016), image segmentation (Baumgartner et al., 2017, 2019; Dou et al., 2020a), and…”
BackgroundSamide, A. P., Stoean, R., Stoean, C., Tutunaru, B., Grecu, R., & Cioateră, N. (2019) · Coatings · doi:10.3390/coatings9020105 · Cited by 28
Medical Image Segmentation Using Deep Learning
López‐Linares, K., Ocaña, M. I. G., Lete, N., Ballester, M. Á. G., & Macía, I. (2019) · Intelligent systems reference library · doi:10.1007/978-3-030-32606-7_2 · Cited by 29
To Aggregate or Not? Learning with Separate Noisy Labels
Wei, J., Zhu, Z., Luo, T., Amid, E., Kumar, A., & Liu, Y. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599522 · Cited by 42
V‐Awake: A Visual Analytics Approach for Correcting Sleep Predictions from Deep Learning Models
Caballero, H. S. G., Westenberg, M. A., Gebre, B. G., & van Wijk, J. J. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13667 · Cited by 26
A simplified cluster model and a tool adapted for collaborative labeling of lung cancer CT scans
Morozov, S. P., Gombolevskiy, V. А., Elizarov, A. B., Гусев, М. А., Novik, V. P., Prokudaylo, S., Bardin, A. S., Popov, E. V., Ledikhova, N. V., Chernina, V., et al. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106111 · Cited by 24
Zhang, Y., Chen, S., Wang, Y., Li, J., Xu, K., Chen, J., & Zhao, J. (2023) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-023-04838-4 · Cited by 26
“Many scholars have applied DL methods based on digital pathology to clinical diagnostic and therapeutic tasks, such as extraction of clinical biological structure information, including cellular level (Albarqouni et al. 2016), and glandular level (Fleming et al.”
BackgroundDeep Learning in Maritime Autonomous Surface Ships: Current Development and Challenges
Ye, J., Li, C., Wen, W., Zhou, R., & Reppa, V. (2023) · Journal of Marine Science and Application · doi:10.1007/s11804-023-00367-1 · Cited by 28
“Nowadays, in addition to the above fields, DL has also been applied to fields such as medical science (Albarqouni et al., 2016), praxeology (Phan et al., 2017), transportation (Lv et al., 2015).”
BackgroundAn Attention Mechanism for Neural Answer Selection Using a Combined Global and Local View
Bachrach, Y., Žukov-Gregorič, A., Coope, S., Tovell, E., Maksak, B., Rodríguez, J. D., McMurtie, C., & Bordbar, M. (2017) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2017.00072 · Cited by 24
“Deep learning methods, which have recently shown great success in many domains including image classification and annotation [16, 17, 18], multi-annotator data fusion [19, 20], NLP and conversational models [21, 22, 23, 24, 25] and speech recognition [19, 21], have also been successfully applied to question answering…”
BackgroundBiomedical imaging and analysis through deep learning
Drukker, K., Yan, P., Sibley, A., & Wang, G. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-821259-2.00004-1 · Cited by 24
Deep learning‐based automated mitosis detection in histopathology images for breast cancer grading
Mathew, T., Ajith, B., Kini, J. R., & Rajan, J. (2022) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22703 · Cited by 26
“Considering the unavailability of large datasets for deep learning, Albarqouni et al. 28 proposed a framework by incorporating a crowd-sourcing layer called AggNet into CNN. Crowd-sourced image annotations are used to train the proposed CNN model.”
MethodRecent Applications of Artificial Intelligence in Early Cancer Detection
Khanam, N., & Kumar, R. (2022) · Current Medicinal Chemistry · doi:10.2174/0929867329666220222154733 · Cited by 24
E-Tanh: a novel activation function for image processing neural network models
Kalaiselvi, T., Padmapriya, S., Somasundaram, K., & Praveenkumar, S. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07245-x · Cited by 23
“…and classification [19], speech recognition [20, 21], segmentation [22, 23], understanding and describing a scenario [24, 25], machine translation [26, 27], speech systems [28, 29], cancer detection [30–32] biometrics [33, 34], forecasting weather [35, 36], automatic self-driving cars [37, 38].”
BackgroundYadav, D. P., Kumar, D., Jalal, A. S., Kumar, A., Singh, K. U., & Shah, M. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-44210-7 · Cited by 23
Chest X-Ray Images Based Automated Detection of Pneumonia Using Transfer Learning and CNN
Thakur, S., Goplani, Y., Arora, S., Upadhyay, R., & Sharma, G. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-4992-2_31 · Cited by 23
Lahiani, A., Gildenblat, J., Klaman, I., Navab, N., & Klaiman, E. (2019) · IET Image Processing · doi:10.1049/iet-ipr.2018.6513 · Cited by 26
Self-citationMehmood, M., Abbasi, S. H., Aurangzeb, K., Majeed, M. F., Anwar, M. S., & Alhussein, M. A. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1225490 · Cited by 22
Jenitta, A., & Ravindran, R. S. (2017) · Journal of Medical Systems · doi:10.1007/s10916-017-0799-z · Cited by 23
Scalable and efficient learning from crowds with Gaussian processes
Morales-Álvarez, P., Ruiz, P., Santos‐Rodríguez, R., Molina, R., & Katsaggelos, A. K. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.12.008 · Cited by 23
Scalable Variational Gaussian Processes for Crowdsourcing: Glitch Detection in LIGO
Morales-Álvarez, P., Ruiz, P., Coughlin, S. B., Molina, R., & Katsaggelos, A. K. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3025390 · Cited by 25
A 2021 update on cancer image analytics with deep learning
Kurian, N. C., Sethi, A., Konduru, A. R., Mahajan, A., & Rane, S. U. (2021) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1410 · Cited by 24
“…for tumor localization (Wang, Khosla, Gargeya, Irshad, & Beck, 2016), nucleus counting (Sirinukunwattana et al., 2016), and mitotic figure counting (Albarqouni et al., 2016) in pathology, as well as detecting lesions in mammograms (Al-masni et al., 2018), skin lesions in clinical images (Ünver &…”
BackgroundWitowski, J. S., Choi, J., Jeon, S., Kim, D., Chung, J. W., Conklin, J., Longo, M. G. F., Succi, M. D., & Do, S. (2021) · Blockchain in Healthcare Today · doi:10.30953/bhty.v4.176 · Cited by 23
“A s the field of supervised machine learning (ML) and artificial intelligence (AI) becomes more mature, researchers working on medical ML or AI aim to integrate more data to improve their models, as opposed to merely changes in the algorithm architecture (1–3).”
BackgroundEmploying Weak Annotations for Medical Image Analysis ProblemsPreprint
Rajchl, M., Koch, L. M., Ledig, C., Passerat‐Palmbach, J., Misawa, K., Mori, K., & Rueckert, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06297 · Cited by 24
“, 2016b) or crowd sourcing platforms (Maier-Hein et al., 2014; Albarqouni et al., 2016)”
BackgroundSurgical Data Science: A Consensus PerspectivePreprint
Maier‐Hein, L., Eisenmann, M., Feldmann, C., Feußner, H., Forestier, G., Giannarou, S., Gibaud, B., Hager, G. D., Hashizume, M., Katić, D., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03184 · Cited by 26
“3) Large-scale data annotation: Methods to facilitate large-scale data annotation, e.g. based on concepts of crowd-sourcing [92], [93], expert data augmentation [94] or self-supervised learning [95].”
BackgroundSelf-citationShen, Y., Li, H., Sun, C., Ji, H., Zhang, D., Hu, K., Tang, Y., Chen, Y., Wei, Z., & Lv, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01014-x · Cited by 21
“This can pose challenges when applying models trained on professional images to non-professional ones 22,23 .”
MethodExtending Label Aggregation Models with a Gaussian Process to Denoise Crowdsourcing Labels
Li, D., & de Rijke, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591685 · Cited by 22
“[1] handle label aggregation directly as part of the learning process of the convolutional neural network via additional crowdsourcing layer.”
BackgroundClassification of diseases from CT images using LSTM-based CNN
Chowdhury, S. R., Khare, Y., & Mazumdar, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-96129-5.00008-1 · Cited by 21
Smart IoT in Breast Cancer Detection Using Optimal Deep Learning
Majji, R., Prakash, P. G. O., Rajeswari, R., & Cristin, R. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-023-00834-9 · Cited by 21
“Deep learning is a promising field where artificial intelligence (AI) and machine learning utilize multiple non-linear processing layers for learning the features from information [17, 23].”
BackgroundWan, T., Zhang, W., Zhu, M., Chen, J., Achim, A., & Qin, Z. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.008 · Cited by 22
Power Distribution Networks Load Forecasting Using Deep Belief Networks: The South African Case
Motepe, S., Hasan, A. N., Twala, B., & Stopforth, R. (2019) · IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT) · doi:10.1109/jeeit.2019.8717434 · Cited by 23
End-to-End Weak SupervisionPreprint
Cachay, S. R., Boecking, B., & Dubrawski, A. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02233 · Cited by 23
Xue, H., Wang, S.-Y., Cui, L., & Hong, K. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20436 · Cited by 20
Wang, S., Zheng, Z., Wang, X., Zhang, Q., & Liu, Z. (2024) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3413732 · Cited by 21
“Albarqouni et al. [17] introduced a novel concept by integrating a crowdsourcing layer that enables direct crowd participation in data aggregation, thus incorporating them into the convolutional neural network learning process.”
Automated diagnosis of neonatal encephalopathy on aEEG using deep neural networks
Wang, J., Ju, R., Chen, Y., Liu, G., & Zhang, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.01.057 · Cited by 22
Truth Discovery in Sequence Labels from Crowds
Sabetpour, N., Kulkarni, A., Xie, S., & Li, Q. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00065 · Cited by 24
A Review and Computational Analysis of Breast Cancer Using Different Machine Learning Techniques
Nemade, V., Pathak, S. K., Dubey, A. K., & Barhate, D. (2022) · International Journal of Emerging Technology and Advanced Engineering · doi:10.46338/ijetae0322_13 · Cited by 22
Bioinformatics and Medicine in the Era of Deep Learning
Bacciu, D., Lisboa, P. J. G., Martín‐Guerrero, J. D., Stoean, R., & Vellido, A. (2018) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.48550/arxiv.1802.09791 · Cited by 24
“CNN modelling may target images from histopathology [34, 35], CT scans [36, 37], or MRI [38], to name a few.”
Bibliometric analysis of the application of deep learning in cancer from 2015 to 2023
Wang, R., Huang, S., Wang, P., Shi, X., Li, S., Ye, Y., Zhang, W., Shi, L., Zhou, X., & Tang, X. (2024) · Cancer Imaging · doi:10.1186/s40644-024-00737-0 · Cited by 19
“In 2016, Albarqouni proposed an additional crowdsourcing layer ( AggNet ) for mitosis of breast cancer histological images, the data aggregation is processed directly through the AggNet and as part of CNNs [37].”
Deep Learning for Shot Classification in Gynecologic Surgery Videos
Petscharnig, S., & Schoeffmann, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-51811-4_57 · Cited by 21
“The importance of CNNs in medical applications is also apparent from their use within other applications such as nucleus segmentation [11], polyp detection in colonoscopy videos [6], microcalcification detection in digital breast tomosynthesis [8], mitosis detection in breast cancer histology [1], and short–term breas…”
MethodDeep Learning and Computer-Aided Diagnosis for Medical Image Processing: A Personal Perspective
Summers, R. M. (2017) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-42999-1_1 · Cited by 20
Segmentation of Liver Lesions Without Contrast Agents With Radiomics-Guided Densely UNet-Nested GAN
Xiao, X., Qiang, Y., Zhao, J., Yang, X., & Yang, X. (2020) · IEEE Access · doi:10.1109/access.2020.3047429 · Cited by 21
Classification of myositis from muscle ultrasound images using deep learning
Uçar, E. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.103277 · Cited by 19
Automated knowledge-assisted mitosis cells detection framework in breast histopathology images
Tan, X. J., Mustafa, N., Mashor, . Y., & Rahman, K. S. A. (2021) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2022081 · Cited by 19
Cardiomegaly Detection using Deep Convolutional Neural Network with U-NetPreprint
Sarpotdar, S. S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.11515 · Cited by 19
“Deep learning-based algorithms have been used to improve image classification accuracy [6] [7].”
MethodTransferring Annotator- and Instance-Dependent Transition Matrix for Learning From Crowds
Li, S., Xia, X., Deng, J., Ge, S., & Liu, T. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3388209 · Cited by 25
“Then, by considering classifier learning, the parameters of classifier and instance-independent transition matrices are estimated jointly by an EM algorithm [29], [30] or in an end-to-end fashion [58].”
BackgroundMethodResultInfluentialGamification in Open and Distance Learning: A Systematic Review
Sümer, M., & Aydın, C. H. (2018) · Springer eBooks · doi:10.1007/978-3-319-17727-4_115-1 · Cited by 20
Breast Cancer Prediction Using Machine Learning Techniques
Apoorva, V., Yogish, H. K., & Chayadevi, M. L. (2021) · Atlantis Highlights in Computer Sciences/Atlantis highlights in computer sciences · doi:10.2991/ahis.k.210913.043 · Cited by 22
“Previously, a group of researchers concentrated on engaging image examination to discriminate between chest illnesses for separating the dangerous developments have infected past the chest, numerous organs and lymph nodes in the vicinity [3-5], and cell science [6-8] utilising unique but little datasets derived from c…”
BackgroundSonar Image Target Detection Based on Deep Learning
Yu, S. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/5294151 · Cited by 20
A deep learning based approach to detect IDC in histopathology images
Gupta, I., Nayak, S. R., Gupta, S., Singh, S., Verma, K., Gupta, A., & Prakash, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11853-5 · Cited by 18
Son, H., Lee, S., Kim, K., Koo, K.-I., & Hwang, C. H. (2022) · Scientific Reports · doi:10.1038/s41598-022-19204-6 · Cited by 20
Multi-Instance Multi-Stage Deep Learning for Medical Image Recognition
Yan, Z., Zhan, Y., Zhang, S., Metaxas, D., & Zhou, X. S. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-810408-8.00006-7 · Cited by 20
Moon, A., Smith, G. H., Kong, J., Rogers, T. E., Ellis, C. L., & Farris, A. B. (2017) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-017-2260-6 · Cited by 18
GT-Net: A Deep Learning Network for Gastric Tumor Diagnosis
Li, Y., Xie, X., Liu, S., Li, X., & Shen, L. (2018) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai.2018.00014 · Cited by 17
“Albarqouni developed a novel deep learning network, namely AggNet, for the mitosis detection in breast cancer histology images [10].”
Modern convolutional object detectors for nuclei detection on pleural effusion cytology images
Baykal, E., Doğan, H., Erçin, M. E., Ersöz, Ş., & Ekіncі, M. (2019) · Multimedia Tools and Applications · doi:10.1007/s11042-019-7461-3 · Cited by 18
Truth Inference With a Deep Clustering-Based Aggregation Model
Yin, L., Liu, Y., Zhang, W., & Yu, Y. H. (2020) · IEEE Access · doi:10.1109/access.2020.2964484 · Cited by 17
“Such an algorithm not only models the relationship between true labels and source labels but also models the relationship between true labels and object features [1], [3], [11], [30]–[33], [44].”
BackgroundMethodInfluentialLiu, L., Chen, X., & Wong, K. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab236 · Cited by 19
Huang, H., Zhou, G., Liu, X., Deng, L., Wu, C., Zhang, D., & Liu, H. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac294 · Cited by 17
OpSeF: Open Source Python Framework for Collaborative Instance Segmentation of Bioimages
Rasse, T. M., Hollandi, R., & Horváth, P. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.558880 · Cited by 20
Sarhan, M. H., Albarqouni, S., Yiğitsoy, M., Navab, N., & Eslami, A. (2020) · IET Image Processing · doi:10.1049/iet-ipr.2019.0804 · Cited by 17
“Another way to counter imbalance is to oversample the under-represented class by balancing the ratios of the classes in each training batch [20].”
BackgroundSelf-citationHaq, I., Gong, Z., Liang, H., Zhang, W., Khan, R., Gu, L., Eils, R., Kang, Y., & Huang, B. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105708 · Cited by 17
Zhao, Y., & Zhou, Y. (2021) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.20220501.oa1 · Cited by 16
Clickstream Analysis for Crowd-Based Object Segmentation with Confidence
Heim, E., Seitel, A., Andrulis, J., Isensee, F., Stock, C., Roß, T., & Maier-Hein, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2777967 · Cited by 19
“Crowdsourcing has become popular in this context as it is based on outsourcing cognitive tasks to many anonymous, untrained individual workers from an online community [5], [6].”
BackgroundKumar, S. N., Fred, A. L., Padmanabhan, P., Gulyás, B. Z., Kumar, H. A., & Miriam, L. R. J. (2020) · Studies in computational intelligence · doi:10.1007/978-981-15-6321-8_3 · Cited by 17
Multi-scale attention U-net for segmenting clinical target volume in graves’ ophthalmopathy
Hu, J., Song, Y., Zhang, L., Bai, S., & Zhang, Y. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.11.028 · Cited by 16
Large-scale gastric cancer screening and localization using multi-task deep neural network
Yu, H., Zhang, X., Song, L., Jiang, L., Huang, X., Chen, W., Zhang, C., Li, J., Yang, J., Hu, Z., et al. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.006 · Cited by 16
Wu, H., Ma, T., Wu, L., Xu, F., & Ji, S. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3460865 · Cited by 21
“Besides, there are also some deep learning models that jointly learn a classifier together with the label aggregation model [1, 4, 9, 37].”
BackgroundAdversarial Learning from Crowds
Chen, P., Sun, H., Yang, Y., & Chen, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20467 · Cited by 16
“Alternatively, more recently proposed one-stage meth-ods (Luo et al. 2018; Yang et al. 2018a) such as CrowdLayer (CL) (Rodrigues and Pereira 2018) and AggNet (Albarqouni and Baur 2016) simultaneously infer the true labels while learning the parameters of the deep neural network and the confusion matrices of annotators.”
BackgroundMethodInfluentialHuk, M. (2016) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-169134 · Cited by 17
Learning from Imperfect AnnotationsPreprint
Platanios, E. A., Al-Shedivat, M., Xing, E. P., & Mitchell, T. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03473 · Cited by 20
“…from model training, although there have been a few notable recent attempts to integrate label aggregation directly into deep neural architectures (sometimes in the form of a layer) in the context of medical imaging (Albarqouni et al., 2016) or natural language tasks (Nguyen et al., 2017).”
BackgroundSolution to overcome the sparsity issue of annotated data in medical domain
Pujitha, A. K., & Sivaswamy, J. (2018) · CAAI Transactions on Intelligence Technology · doi:10.1049/trit.2018.1010 · Cited by 15
“A convolutional neural network (CNN) was trained in [8] and the crowdsourced mitosis candidates (in a patch of size 33 × 33 ) were merged with an aggregation layer for updating the model.”
MethodApplication of Computer Vision and Deep Learning in Breast Cancer Assisted Diagnosis
Gu, Y., & Jiayao, Y. (2019) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3310986.3311010 · Cited by 19
“obtained the labeling of a large number of non-medical experts' pathological slices of breast cancer without labeling by "crowdsourcing" technology, and sent these labeling to a specially designed CNN network for calibration through target detection technology[30] This method of putting labels from "crowdsourcing" int…”
MethodGao, E., Jiang, H., Zhou, Z., Yang, C., Chen, M., Zhu, W., Shi, F., Chen, X., Zheng, J., Bian, Y., et al. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106228 · Cited by 15
Fraud Detection under Multi-Sourced Extremely Noisy Annotations
Zhang, C., Wang, Q., Liu, T., Lu, X., Hong, J., Han, B., & Gong, C. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482433 · Cited by 15
“The second category is referred as the joint approach, which simultaneously infers true labels and trains the classifier [2, 30, 42].”
BackgroundThapa, V., Galande, A. S., Ram, G. H. P., & John, R. (2024) · Journal of Biomedical Optics · doi:10.1117/1.jbo.29.1.016010 · Cited by 14
“54 This property makes it suitable for various tasks, such as classification, 55 cancer detection, 56 and segmentation.”
MethodCoupled Confusion Correction: Learning from Crowds with Sparse Annotations
Zhang, H., Li, S., Zeng, D., Yan, C. C., & Ge, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29613 · Cited by 25
Learning Self-calibrated Optic Disc and Cup Segmentation from Multi-rater Annotations
Wu, J., Fang, H., Wang, Z., Yang, D., Yang, Y., Shang, F., Zhou, W., & Xu, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16434-7_59 · Cited by 14
Metadata
- Authors
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N.
- Affiliations
- German Center for Neurodegenerative Diseases (DE)
- Technical University of Munich (DE)
- University of Oxford (GB)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1313-1321
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2339885376
- Cited by
- 606
- References
- 38
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 59 · 2023: 59 · 2022: 60 · 2021: 92 · 2020: 89 · 2019: 80 · 2018: 73 · 2017: 46 · 2016: 10
- FWCI
- 57.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- DZNE Pub (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- mitosis detection, breast cancer histopathology, crowdsourced annotation, noisy annotations, data aggregation, convolutional neural networks, CrowdFlower
- MeSH terms
- Machine Learning, Breast Neoplasms, Female, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Mitosis, Neural Networks, Computer, Video Games, Internet, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.