AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Solving the recurrent issue of inadequate labeling during the analysis of medical images can be achieved by feeding the crowd-sourced input labels into the CNN.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
In the citing paper
“Albarquoni et al. [294] investigated the problem of insufficient labeling. They crowd-sourced the actual mitoses labeling in the histology images of breast cancer (from amateurs online).”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
BackgroundMethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
(2010) · International Conference on Machine Learning · Cited by 13,995
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
MethodMatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 1×
“vNet [34] and conducted on an Intel i7 machine with a GeForce”
MethodTowards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858 · cited 1×
“C ROWDSOURCING is a type of participative online activity in which an individual, an institution, a non-profit organization or a company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via flexible open call, the voluntary undertaking of a task [1].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Being inspired by prominent previous work in this field [18], [26], we define the specific contribution of our own work as: i) Learning of a multi-scale CNN model for mitosis detection, ii) Incorporation of aggregation schemes into CNN layers, and iii) Augmentation and retraining of the CNN model with crowd's annotati…”
Method
Research works printed but not linked: 1
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
Cited by
Items 401–500 of 613
Collazo, C., Vargas, I., Cara, B. D., Weinheimer, C. J., Grabau, R. P., Goldgof, D. B., Hall, L., Wickline, S. A., & Pan, H. (2024) · Bioengineering · doi:10.3390/bioengineering11050434 · Cited by 5
Data Tagging in Medical Images: A Survey of the State-of-Art
Chaki, J., & Dey, N. (2020) · Current Medical Imaging Formerly Current Medical Imaging Reviews · doi:10.2174/1573405616666200218130043 · Cited by 5
Correlated Chained Gaussian Processes for Datasets With Multiple Annotators
Gil-González, J., Giraldo, J.-J., Álvarez-Meza, A. M., Orozco, A. A., & Álvarez, M. A. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3116943 · Cited by 5
“Furthermore, some works have addressed the multilabeler problem using deep learning approaches typically including an extra layer that codes the annotators’ information [17]–[19].”
BackgroundMayerich, D. M., Sun, R., & Guo, J. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-821049-9.00015-0 · Cited by 6
Probabilistic fusion of crowds and experts for the search of gravitational waves
Ruiz, P., Morales-Álvarez, P., Coughlin, S. B., Molina, R., & Katsaggelos, A. K. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110183 · Cited by 5
Li, X., Heirman, C. C., Rickard, A. G., Sotolongo, G., Castillo, R. J., Adanlawo, T., Everitt, J. I., Hodgin, J. B., Watts, T. L., Janowczyk, A. R., et al. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1451261 · Cited by 5
Deep Learning for Clinical Image Analysis
Thangarasan, T., Potnuru, M., Ramesh, P. S., Nandhini, T. J., Surendran, S., & Balaji, A. (2024) · International Conference on Communication, Computing and Energy Efficient Technologies (I3CEET) · doi:10.1109/i3ceet61722.2024.10994101 · Cited by 7
Qin, K. X., Ahmad, N., Soomro, K., Yogaiswaaran, S., Singogo, T. D., & Rasheed, A. J. (2024) · Quest International Journal of Medical and Health Sciences · doi:10.5281/zenodo.13139999 · Cited by 5
Landscape of Big Medical Data: A Pragmatic Survey on Prioritized Tasks
Zhang, Z., Gao, W., Zhang, F., Huang, Y., Dai, S., Fan, F., Zhan, J., Du, M., Yin, S., Xiong, L., et al. (2019) · IEEE Access · doi:10.1109/access.2019.2891948 · Cited by 5
Aggregating and Learning from Multiple Annotators
Paun, S., & Simpson, E. (2021) · doi:10.18653/v1/2021.eacl-tutorials.2 · Cited by 5
“…their noise distillation capabilities, as their learning processes also produce aggregated labels that leverage not only coder annotation patterns but also the knowledge of the task accumulated by the model (Cao et al., 2018; Rodrigues and Pereira, 2018; Albarqouni et al., 2016; Chu et al., 2020).”
BackgroundFast and Interpretable Deep Learning Pipeline for Breast Cancer Recognition
Bonyani, M., Yeganli, F., & Yeganli, S. F. (2022) · Medical Technologies Congress (TIPTEKNO) · doi:10.1109/tiptekno56568.2022.9960227 · Cited by 5
“[9] developed a DL model that utilizes crowd annotations (non-expert) and incorporates it into the CNN learning process via an additional crowdsourcing layer to improve model performance.”
MethodHeinisch, P., Orlikowski, M., Romberg, J., & Cimiano, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.687 · Cited by 20
“Most work in this area (Raykar et al., 2010; Albarqouni et al., 2016; Guan et al., 2018; Rodrigues and Pereira, 2018) argues for training on non-aggregated labels as a way to deal with varying annotator reliability to better derive the correct labels.”
BackgroundXue, Y., Ray, N., Hugh, J. C., & Bigras, G. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296696 · Cited by 5
“Expectation maximization has been utilized within the DL framework in an end-to-end fashion for mitosis detection [3].”
MethodStoean, R., Stoean, C., & Samide, A. P. (2018) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc.2018.00065 · Cited by 5
“However, unlike recent microscopy image interpretation by CNN in these two fields [27], [28], [29], [30], [31], [32], the physical sciences have not benefited much from the promise of the new star of DL.”
BackgroundLabelling Training Samples Using Crowdsourcing Annotation for Recommendation
Wang, Q., Zhang, M., Tao, T., & Sheng, V. S. (2020) · Complexity · doi:10.1155/2020/1670483 · Cited by 5
“Our multiple experiments are conducted on MTurk, which is a welcome crowdsourcing marketplace supporting crowdsourced execution of Human Intelligence Tasks (HITs) [12].”
BackgroundMethodCrowd R-CNN: An Object Detection Model Utilizing Crowdsourced Labels
Hu, Y., & Song, M. (2019) · International Conference on Vision, Image and Signal Processing (ICVISP) · doi:10.1145/3387168.3387180 · Cited by 10
“� In our aggregation layers, we integrate the method initially proposed by [31], showing a good performance in many applications [27].”
BackgroundMethodArtificial intelligence methods for predictive image-based grading of human cancers
Fernández, G., Madduri, A. S., Marami, B., Prastawa, M. W., Scott, R., Zeineh, J., & Donovan, M. J. (2020) · Elsevier eBooks · doi:10.1016/b978-0-323-67538-3.00009-9 · Cited by 6
Deep and Confident Image Analysis for Disease Detection
Wambura, S. M., & Li, H. (2020) · International Conference on Video, Signal and Image Processing (VSIP) · doi:10.1145/3442705.3442720 · Cited by 8
“…and pneumonia diseases, lung diseases such as TB, pneumonia, and lung cancer [19], crowds for mitosis detection in breast cancer histology images [1], Breast Cancer classification from histopathological images with inception Recurrent residual CNN [2], image analysis and ML for detecting malaria…”
BackgroundShah, R., Nunes, B. A. A., Gleason, T., Fletcher, W., Banaga, J., Sweetwood, K., Ye, A., Patel, R., McGill, K., Link, T. M., et al. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00662-3 · Cited by 5
RECENT CNN-BASED TECHNIQUES FOR BREAST CANCER HISTOLOGY IMAGE CLASSIFICATION
Karuppasamy, A., Abdesselam, A., Zidoum, H. M., & Al-Bahri, M. (2022) · The Journal of Engineering Research [TJER] · doi:10.53540/tjer.vol19iss1pp41-53 · Cited by 5
“ArunaDevi Karuppasamy, A. Abdesselam, R. Hedjam, H. Zidoum , and Maiya Al-Bahri CNN-based methods, including those addressing histopathology image classification, usually face two major challenges: 1)- there is no systematic approach for designing an appropriate CNN architecture and 2)- there is a need for large-scale…”
BackgroundMethodInfluentialDas, S., Jayakumar, J., Banerjee, S., Ramaswamy, J., Vangala, V., Ram, K., & Mitra, P. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/252247 · Cited by 5
“One method for mitosis detection [4] is based on deep learning which performs data aggregation using convolutional neural networks (CNN) via additional crowdsourcing layer (AggNet).”
MethodLearn to Estimate Labels Uncertainty for Quality AssurancePreprint
Tomczack, A., Navab, N., & Albarqouni, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08058 · Cited by 5
“One approach to the problem would be modeling inter-observer variability through crowd-sourcing techniques, as suggested in [1, 7], and interpret the variance of multiple predictions as labels uncertainty.”
Self-citationWang, J., Wang, J., Ma, J., Chen, B., Chen, Z., & Zheng, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103812 · Cited by 4
Griffith, J. L., Joseph, J., Jensen, A., Banks, S. A., & Allen, K. D. (2025) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2025.02.787 · Cited by 4
Robust Authentication: Leveraging Hardware Fingerprints and AI to Enhance Security Against Spoofing
Nimmy, K., Jain, K., Sachin, S., Abekaesh, P. A., & Venkitasubramaniam, P. (2025) · IEEE Access · doi:10.1109/access.2025.3569881 · Cited by 4
“It is worth noting that previous studies, such as those by Albarqouni et al. [37], and Celaya et al. [38] provide strong evidence that deep learning models can perform effectively even with datasets as small as 100 images, as well as with larger datasets, depending on the model architecture and…”
Kusuma Sri, M., & Sathees Kumaran, S. (2025) · Journal of Innovative Image Processing · doi:10.36548/jiip.2025.2.002 · Cited by 4
“Convolutional neural networks have addressed these challenges, resulting in improved accuracy [6].”
SwifTree: Interactive Extraction of 3D Trees Supporting Gaming and Crowdsourcing
Huang, M., & Hamarneh, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67534-3_13 · Cited by 4
Akogo, D., & Palmer, X. (2019) · Lecture notes in networks and systems · doi:10.1007/978-3-030-12385-7_13 · Cited by 4
Breast Epithelial Duct Region Segmentation Using Intuitionistic Fuzzy Based Multi-Texture Image Map
Belsare, A. D., Mushrif, M. M., & Pangarkar, M. A. (2017) · IEEE India Conference · doi:10.1109/indicon.2017.8487730 · Cited by 4
Survey on Mitosis Detection for Aggressive Breast Cancer from Histological Images
Hussain, H., Hujran, O., & Nitha, K. P. (2019) · International Conference on Information Management · doi:10.1109/infoman.2019.8714696 · Cited by 4
“By sliding NN detector over images AGGNET [11] - Images annotated by pathologists Image was taken as a whole CNN with additional Aggregatio n Layer.”
MethodDeep Learning for Cancer Cell Detection and Segmentation: A Survey
Hajela, P., Pawar, A., & Ahirrao, S. N. (2018) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon.2018.8745420 · Cited by 4
“Unlike typical supervised methods, which learn a model from ground truth labeled data, learning from crowd annotations is different in the sense that there may be (possibly noisy) multiple labels for the same sample [5].”
BackgroundBreast Cancer Detection using Deep Learning
Amdani, S. Y. (2019) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2019.5443 · Cited by 4
Learning deep networks with crowdsourcing for relevance evaluation
Wu, M., Yin, X., Li, Q., Zhang, J., Feng, X., Cao, Q., & Shen, H. (2020) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-020-01697-2 · Cited by 5
“Generally, deep learning is also considered to used in ground truth inferencing in several works [23–28].”
BackgroundMethodA Survey on Deep Learning in Financial Markets
Zhang, J., Zhai, J., & Wang, H. (2021) · Financial Mathematics and Fintech · doi:10.1007/978-981-15-8373-5_3 · Cited by 6
Convolutional Neuronal Networks for Tumor Regions Detection in Histopathology Images
Benomar, M. L., Settouti, N., Xiao, R., Ambrosetti, D., & Descombes, X. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-73882-2_2 · Cited by 4
Data Poisoning Attacks on Crowdsourcing Learning
Chen, P., Sun, H., & Chen, Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85896-4_14 · Cited by 4
COVID-19 disease identification network based on weakly supervised feature selection
Liu, J., Feng, Q., Miao, Y., He, W., Shi, W., & Jiang, Z. (2023) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2023409 · Cited by 4
Thalakottor, L. A., Shirwaikar, R. D., Pothamsetti, P. T., & Mathews, L. M. (2023) · Critical Reviews in Biomedical Engineering · doi:10.1615/critrevbiomedeng.2023047793 · Cited by 4
Saini, V., Khurana, M., & Challa, R. K. (2023) · International Conference on Circuits, Power and Intelligent Systems (CCPIS) · doi:10.1109/ccpis59145.2023.10291244 · Cited by 4
“These are the pre-trained models which were already trained over million images to perform the batch work [19-20].”
MethodSpicher, N., Wesemeyer, T., & Deserno, T. M. (2023) · Biomedizinische Technik/Biomedical Engineering · doi:10.1515/bmt-2023-0148 · Cited by 4
Visualizing Wellness: The Impact of Deep Learning on Healthcare Image Analysis
Sinha, D. K., Naval, P., & Fiza, S. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10465825 · Cited by 4
“[9] The most popular models are recurrent neural networks and convolutional neural networks.”
BackgroundNancy, P., Rajeshram, V., Kumar, G. S., & Dhivya, P. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106498 · Cited by 4
KD-Crowd: a knowledge distillation framework for learning from crowds
Li, S.-Y., Zheng, Y., Shi, Y., Huang, S.-J., & Chen, S. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-023-3578-7 · Cited by 4
Role of Task Complexity and Training in Crowdsourced Image Annotation
Schaadt, N. S., Grote, A., Forestier, G., Wemmert, C., & Feuerhake, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00949-6_6 · Cited by 4
“Crowdsourcing (CS) in digital pathology has been largely limited to less complex tasks such as identification of cancer cells [11,6], scoring of cell nuclei based on immunohistochemistry (IHC) [2,11,4,6], malaria diagnostics [9], and creation of training sets for convolutional neural networks [1,5].”
MethodFine-Tuning Deep Learning by Crowd Participation
Albarqouni, S. (2018) · IEEE Pulse · doi:10.1109/mpul.2018.2866356 · Cited by 4
Self-citationUSTAR: Online Multimodal Embedding for Modeling User-Guided Spatiotemporal Activity
Silva, A., Karunasekera, S. A., Leckie, C. A., & Luo, L. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9005461 · Cited by 4
“ork (compared with USTAR in SectionII-A) that learns embeddings in an online fashion. Online learning has been previously used in other domains to model the temporal evolution of words [25], diseases [26] and networks [27,28]. However, USTAR differs from the above studies by being specific to the task of spatiotemporal…”
MethodJoshi, S. A., Bongale, A. M., & Bongale, A. M. (2021) · Lincoln (University of Nebraska) · Cited by 4
Kimeswenger, S., Rumetshofer, E., Hofmarcher, M., Tschandl, P., Kittler, H., Hochreiter, S., Hötzenecker, W., & Klambauer, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06616 · Cited by 4
“Recent attempts in training CNNs on histopathologic images frequently avoid the ultra-high resolution problem by sampling random image patches from the full image (Cruz-Roa et al., 2013; Albarqouni et al., 2016; Janowczyk and Madabhushi, 2016), which maintains the weak labelling problem.”
BackgroundOpSeF: Open source Python framework for collaborative instance segmentation of bioimagesPreprint
Rasse, T. M., Hollandi, R., & Horváth, P. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.29.068023 · Cited by 5
Multi-task Ensembles with Crowdsourced Features Improve Skin Lesion DiagnosisPreprint
Raumanns, R., Contar, E. K., Schouten, G., & Cheplygina, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.14745 · Cited by 4
“For example, in [2] crowds alone do not provide labels of sufficient quality when asked to detect mitosis in histopathology images.”
BackgroundFelipe, G. Z., Teixeira, L., Pereira, R. M., Zanoni, J. N., de Souza, S. R. G., Nanni, L., Cavalcanti, G. D. C., & Costa, Y. M. G. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-11114-y · Cited by 3
A deep learning model for multiclass lung cancer classification using multimodal data fusion
Lin, W., Cheng, X., Wang, Y., Zhou, S., Zhong, W., Chen, W., Song, S., Gan, S., Cheng, Y., Chen, D., et al. (2025) · Discover Oncology · doi:10.1007/s12672-025-04168-6 · Cited by 3
“In recent years, classification and diagnosis techniques based on pathological images have made significant progress [3, 4].”
An empirical study for breast cancer detection using MRI images
Jaincy, D. E. M., & Pattabiraman, V. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109640 · Cited by 3
Task design for crowdsourced glioma cell annotation in microscopy images
Schwarze, S., Schaadt, N. S., Sobotta, V. M. G., Spicher, N., Skripuletz, T., Esmaeilzadeh, M., Krauss, J. K., Hartmann, C. J., Deserno, T. M., & Feuerhake, F. (2024) · Scientific Reports · doi:10.1038/s41598-024-51995-8 · Cited by 3
Crowdsourced Annotations as an Additional Form of Data Augmentation for CAD Development
Kandala, P. A., & Sivaswamy, J. (2017) · Asian Conference on Pattern Recognition · doi:10.1109/acpr.2017.6 · Cited by 3
“Methods for merging ranges from simple Majority Voting (MV) [6] to a stochastic modeling of the crowdsourced information using Expectation Maximization [4] and introducing an aggregation layer in a CNN [8].”
BackgroundMethodInfluentialMinimum Precision Requirements for Deep Learning with Biomedical Datasets
Sakr, C., & Shanbhag, N. R. (2018) · IEEE Biomedical Circuits and Systems Conference (BioCAS) · doi:10.1109/biocas.2018.8584732 · Cited by 3
“Consequently, deep learning is currently being employed for several biomedical applications such as medical imaging [5], biological anomaly and pathology detection [6], and early diagnosis of diseases [7], to name a few.”
MethodOn the evaluation and selection of classifier learning algorithms with crowdsourced data
Urkullu, A., Perez, A., & Calvo, B. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.02.019 · Cited by 3
Improvement of Whole-Slide Pathological Image Recognition Method Based on Deep Learning
Ma, X., Liu, H., Niu, Y., Zhang, C., & Liu, D. (2018) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid.2018.10163 · Cited by 3
Classification of Brain Tumor from Magnetic Resonance Imaging using Convolutional Neural Networks
Zyad, M.-A., Gouskir, M., & Bouikhalene, B. (2019) · International Journal of Advanced Science and Technology · doi:10.33832/ijast.2019.126.04 · Cited by 3
A Study on Deep Learning for Breast Cancer Detection in Histopathological Images
Singh, O. V., Choudhary, P., & Thongam, K. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-4018-9_4 · Cited by 3
Arellano-Espitia, F., & Soto, L. R. (2020) · IntechOpen eBooks · doi:10.5772/intechopen.89570 · Cited by 3
CrowdDetective: Wisdom of the Crowds for Detecting Abnormalities in Medical Scans
Cheplygina, V. (2020) · Journal of Trial and Error · doi:10.36850/rga1 · Cited by 3
“Only [27] uses the labels to train algorithms, but collects labels and does not address the MIL scenario.”
BackgroundMethodCrowdsourcing airway annotations in chest computed tomography images
Cheplygina, V., Perez‐Rovira, A., Kuo, W., Tiddens, H. A. W. M., & de Bruijne, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0249580 · Cited by 3
“For example, combinations of expert and crowdsourced labels have shown to be more effective in cases where the crowdsourced labels alone were not sufficient [14].”
Process of Medical Dataset Construction for Machine Learning - Multifield Study and Guidelines
Cychnerski, J., & Dziubich, T. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85082-1_20 · Cited by 4
Semi-supervised Yolo Network for Induced Pluripotent Stem Cells Detection
Wang, X., Liao, J., Yue, G., He, L., Li, M., Liang, E., Wang, T., Zhou, G., & Lei, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87358-5_65 · Cited by 3
Deep Learning Techniques For Improving Breast Cancer Detection And Diagnosis
Abed, A. H. (2022) · International Journal of Advanced Networking and Applications · doi:10.35444/ijana.2022.13606 · Cited by 18
Learning From Crowds With Contrastive Representation
Yang, H., Li, X., & Witold Pedrycz, C. (2023) · IEEE Access · doi:10.1109/access.2023.3269751 · Cited by 3
“The recent one trains the downstream model and annotator parameters simultaneously in an end-to-end fashion [13], [14], [15], [16], [17].”
BackgroundDetermination of bone marrow cell morphology in rat
Shan, J. L., Zhu, G. Y., Su, L. H., Liang, X. S., Fu, J., & Zhang, Z. F. (2023) · Cellular and Molecular Biology · doi:10.14715/cmb/2023.69.11.7 · Cited by 3
“This method has been used to detect mitosis in breast cancer tissue sections (14), skin cancer diagnosis (15), mammography evaluations (16), and cytological classifications in peri - pheral blood (17).”
MethodSun, D., Li, H., Wang, Y., Li, D., Xu, D., & Zhang, Z. (2024) · Current Problems in Cancer · doi:10.1016/j.currproblcancer.2024.101150 · Cited by 3
Using Crowdsourcing for Multi-label Biomedical Compound Figure Annotation
de Herrera, A. G. S., Schaer, R., Antani, S., & Müller, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46976-8_24 · Cited by 3
“A quality control (QC) is needed when using crowdsourcing to ensure the success of the annotation task [13], particularly with medical images where some domain knowledge is very benefitial [1].”
BackgroundComputer-Aided Diagnosis in Chest Radiography with Deep Multi-Instance Learning
Qu, K., Chai, X., Liu, T., Zhang, Y., Leng, B., & Xiong, Z. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70093-9_77 · Cited by 3
Adu, K., Yu, Y., Cai, J., Tattrah, V. D., Ansere, J. A., & Tashi, N. (2021) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-202638 · Cited by 3
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation
Al-Qershi, F., Al‐Qurishi, M., Aksoy, M. S., Faisal, M., & Algabri, M. (2021) · Sensors · doi:10.3390/s21155007 · Cited by 3
“Moreover, many researchers have incorporated crowdsourcing into studies on the COVID-19 pandemic [4], disasters [5], fake news [6], and preprocessing for deep learning applications [7,8].”
BackgroundMatheny, M. E., Ohno-Machado, L., Davis, S. E., & Nemati, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00031-0 · Cited by 3
Is Histopathology Deep Learning Artificial Intelligence the Future of Precision Oncology?
Wagner, V. M. (2024) · Journal of Clinical Oncology · doi:10.1200/jco-24-01271 · Cited by 3
Applications of Computer Vision and Machine Learning in Bioimaging
Argunşah, A. Ö., Erdil, E., & Ünay, D. (2025) · Springer eBooks · doi:10.1007/978-3-031-99410-4_28 · Cited by 3
Building a Deep CNN-BiLSTM Model for Pneumonia Diseases Prediction on Chest X-Ray Images
Bhamare, R., Pandey, V., Sayyed, Z., Kamble, S., Paithanka, S., & Walia, R. (2025) · IEEE International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET) · doi:10.1109/acroset66531.2025.11280621 · Cited by 2
“Using deep learning including but not limited to hybrid models such as Deep CNN - Bi-LSTM further opens the door to better understand and utilize the analysis of standard medical imaging data and patient data, leads to improved timely interventions and thus, improved health outcomes [10]-[12].”
Puvar, P. D., & Prajapati, B. P. (2026) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-025-00649-4 · Cited by 2
“Another innovation incorporates learning from crowds by integrating data aggregation directly into the convolutional neural network through an additional crowdsourcing layer (Albarqouni et al. 2016).”
Semi-Automatic Retrieval of Relevant Segments from Laparoscopic Surgery Videos
Petscharnig, S. (2017) · International Conference on Multimedia Retrieval · doi:10.1145/3078971.3079008 · Cited by 2
“…apparent from their use within other applications such as nucleus segmentation [22], polyp detection in colonoscopy videos [11], microcalcification detection in digital breast tomosynthesis [17], mitosis detection in breast cancer histology [1], and short–term breast cancer risk prediction [14].”
BackgroundClassification of Mitotic Cells: Potentials Beyond the Limits of Small Data Sets
Krappmann, M., Aubreville, M., Maier, A., Bertram, C., & Klopfleisch, R. (2018) · Informatik aktuell · doi:10.1007/978-3-662-56537-7_66 · Cited by 2
“[8] came up with a special type of network that is able process two types of input data.”
BackgroundA Robust Consistency Model of Crowd Workers in Text Labeling Tasks
Al-Qershi, F., Al‐Qurishi, M., Aksoy, M. S., Alrubaian, M., & Imran, M. A. (2020) · IEEE Access · doi:10.1109/access.2020.3022773 · Cited by 2
“Consequently, their research incorporate with the crowdsourcing, for example, responding to Covid-19 pandemic [4] and disasters [5], detecting fake news [6], and deep learning applications [7], [8].”
BackgroundDeep Generative Crowdsourcing Learning with Worker Correlation Utilization
Li, S.-Y., Wei, M., & Huang, S.-J. (2022) · International Journal of Software and Informatics · doi:10.21655/ijsi.1673-7288.00270 · Cited by 2
TshFNA-Examiner: A Nuclei Segmentation and Cancer Assessment Framework for Thyroid Cytology Image
Ke, J., Zhu, J., Yang, X., Zhang, H., Sun, Y., Wang, J. E., Lu, Y., Shen, Y., Liu, S., Jiang, F., et al. (2024) · Journal of Shanghai Jiaotong University (Science) · doi:10.1007/s12204-024-2743-y · Cited by 4
“In the context of pathology, we apply deep learning based on digitized whole slide images (WSIs) to distinguish between normal and malignant tissues, as such, to perform typical diagnostic tasks of tumor segmentation [9-10] , cancer classification [11-12] and abnormality detection [13-14] .”
Kumar, S., & Sharma, S. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-22158-3.00009-0 · Cited by 2
Unlocking AutoML: Enhancing Data with Deep Learning Algorithms for Medical Imaging
Jesus, R., Rodrigues, A. M. M., & Costa, C. M. A. (2024) · Journal of Data and Information Quality · doi:10.1145/3705896 · Cited by 2
“To address the difficulty in producing labeled datasets, Reference [2] proposes a CNN architecture that can be trained on crowdsourced labeled data, achieving an F1-score of 74.2% on the developed algorithm.”
InfluentialMultimodal information capture based truth inference network in crowdsourcing
Han, T., Ding, X., & Fang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126885 · Cited by 2
End-to-end learning via a convolutional neural network for cancer cell line classification
Akogo, D., & Palmer, X. (2019) · Journal of Industry-University Collaboration · doi:10.1108/jiuc-02-2019-002 · Cited by 2
Weighting and Classification of Image Features using Optimization Algorithms
Öztürk, Ş., Özkaya, U., Akdemïr, B., & Seyfi, L. (2018) · International Symposium on Fundamentals of Electrical Engineering · doi:10.1109/isfee.2018.8742479 · Cited by 2
“As a result, artificial intelligence methods have been used in many areas such as various engineering branches [3], medicine [4], production [5], forecasting [6], social media”
MethodViciana, C., & Jones, P. (2020) · Malaysian Sports Journal · doi:10.26480/msj.02.2019.10.13 · Cited by 2
A Pseudo Lesion Generation Method for Deep Learning Based Chest X-Ray Lung Disease Detection
Ai, X., Cao, M., & Li, X. (2021) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie52039.2021.9389924 · Cited by 2
Cheng, C., Chen, W., & Li, A. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/8647298 · Cited by 2
“between the law and themselves, how to deal with their own rights and interests, and whether they will violate traffic rules [23].”
BackgroundJiang, L. (2022) · Mobile Information Systems · doi:10.1155/2022/3356213 · Cited by 2
Deep Separable Convolution Network for Prediction of Lung Diseases from X-rays
Geetha, N., & Sun, J. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0130662 · Cited by 2
“, [13] 110,120 images five different CNN models In terms of execution rate and accuracy, this might be the most efficient DL conceptual model.”
BackgroundInfluentialDiabetic Retinopathy Detection with Weighted Cross-Entropy Loss
Huang, J., Wu, X., Qi, H., Cheng, J.-S., & Zhang, T. (2023) · Chinese Control Conference (CCC) · doi:10.23919/ccc58697.2023.10240083 · Cited by 4
“In recent years, deep learning also has wide applications in medicine and brings great hope to world-wide patients [5, 6].”
BackgroundA regularized CNN approach for detecting cervical cancer
Tanimu, J. J., Hamada, M., Hassan, M., Samson, T. J., & Parah, J. S. (2024) · AIP conference proceedings · doi:10.1063/5.0235521 · Cited by 3
Yi, D., Grøvik, E., Iv, M., Tong, E., Zaharchuk, G., & Rubin, D. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09501 · Cited by 2
“Others have examined the potential for crowdsourcing labels (Albarqouni et al., 2016; Irshad et al., 2014), as done for many computer vision datasets, but medical images often require extensive domain knowledge to achieve accurate annotations.”
BackgroundCollective annotation patterns in learning from crowds
Mena, F., Ñanculef, R., & Valle, C. (2020) · Intelligent Data Analysis · doi:10.3233/ida-200009 · Cited by 1
“Simultaneously, Rodrigues et al. [10] extend [9] to multiple classes, making minor variations to the mathematical formulation.”
InfluentialMager, T., Khademi, S., Siebes, R., van Gemert, J. C., de Boer, V., Löffler, B., & Hein, C. (2023) · Bielefeld University Press eBooks · doi:10.1515/9783839469132-014 · Cited by 1
Mixing Methods: Practical Insights from the Humanities in the Digital Age
Schneider, B., Löffler, B., Mager, T., & Hein, C. (2023) · Bielefeld University Press eBooks · doi:10.14361/9783839469132 · Cited by 1
Object Detection Model for Poultry Diseases Diagnostics
Karim, S., Machuve, D. Z., Mkoba, E. S., & Pandhare, S. M. (2024) · Progress in IS · doi:10.1007/978-3-031-56576-2_9 · Cited by 1
Metadata
- Authors
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N.
- Affiliations
- German Center for Neurodegenerative Diseases (DE)
- Technical University of Munich (DE)
- University of Oxford (GB)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1313-1321
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2339885376
- Cited by
- 606
- References
- 38
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 59 · 2023: 59 · 2022: 60 · 2021: 92 · 2020: 89 · 2019: 80 · 2018: 73 · 2017: 46 · 2016: 10
- FWCI
- 57.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- DZNE Pub (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- mitosis detection, breast cancer histopathology, crowdsourced annotation, noisy annotations, data aggregation, convolutional neural networks, CrowdFlower
- MeSH terms
- Machine Learning, Breast Neoplasms, Female, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Mitosis, Neural Networks, Computer, Video Games, Internet, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.