AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
The lack of publicly available ground-truth data has been identified as the major challenge for transferring recent developments in deep learning to the biomedical imaging domain. Though crowdsourcing has enabled annotation of large scale databases for real world images, its application for biomedical purposes requires a deeper understanding and hence, more precise definition of the actual annotation task. The fact that expert tasks are being outsourced to non-expert users may lead to noisy annotations introducing disagreement between users. Despite being a valuable resource for learning annotation models from crowdsourcing, conventional machine-learning methods may have difficulties dealing with noisy annotations during training. In this manuscript, we present a new concept for learning from crowds that handle data aggregation directly as part of the learning process of the convolutional neural network (CNN) via additional crowdsourcing layer (AggNet). Besides, we present an experimental study on learning from crowds designed to answer the following questions. 1) Can deep CNN be trained with data collected from crowdsourcing? 2) How to adapt the CNN to train on multiple types of annotation datasets (ground truth and crowd-based)? 3) How does the choice of annotation and aggregation affect the accuracy? Our experimental setup involved Annot8, a self-implemented web-platform based on Crowdflower API realizing image annotation tasks for a publicly available biomedical image database. Our results give valuable insights into the functionality of deep CNN learning from crowd annotations and prove the necessity of data aggregation integration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Solving the recurrent issue of inadequate labeling during the analysis of medical images can be achieved by feeding the crowd-sourced input labels into the CNN.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
In the citing paper
“Albarquoni et al. [294] investigated the problem of insufficient labeling. They crowd-sourced the actual mitoses labeling in the histology images of breast cancer (from amateurs online).”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [294]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
BackgroundMethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
(2010) · International Conference on Machine Learning · Cited by 13,995
Large-Scale Machine Learning with Stochastic Gradient Descent
Bottou, L. (2010) · Proceedings of COMPSTAT'2010 · doi:10.1007/978-3-7908-2604-3_16 · Cited by 5,795
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442 · cited 1×
“Indeed, deep learning has advanced the field of computer vision the last few years [19] leading to powerful methods for various applications such as object classification [5], detection [20], segmentation [21], robust regression [22] and depth prediction [23].”
MethodMatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444 · cited 1×
“vNet [34] and conducted on an Intel i7 machine with a GeForce”
MethodTowards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858 · cited 1×
“C ROWDSOURCING is a type of participative online activity in which an individual, an institution, a non-profit organization or a company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via flexible open call, the voluntary undertaking of a task [1].”
BackgroundMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Being inspired by prominent previous work in this field [18], [26], we define the specific contribution of our own work as: i) Learning of a multi-scale CNN model for mitosis detection, ii) Incorporation of aggregation schemes into CNN layers, and iii) Augmentation and retraining of the CNN model with crowd's annotati…”
Method
Research works printed but not linked: 1
- howe, 2006, The rise of crowdsourcing, Wired Mag, 14, 1
Cited by
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293
“Test (n=129 slides and images) RUMC 23 6 50 14 15 2 (1-14) 79”
BackgroundSelf-citationActivation Functions: Comparison of trends in Practice and Research for Deep Learning
Nwankpa, C. E., Ijomah, W., Gachagan, A., & Marshall, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03378 · Cited by 1,240
“to speech 4 recognition [27], [28], segmentation [29], [30], scene understanding and description [31], [32], machine translation [33], [34] test to speech systems [35], [36], cancer detection systems [37], [38], [39], finger print detection [40], [41], weather forecast [42], [43], self-driving cars [44], [45], and othe…”
BackgroundGreenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
Deep Learning Applications in Medical Image Analysis
Ker, J., Wang, L., Rao, J., & Lim, T. (2018) · IEEE Access · doi:10.1109/access.2017.2788044 · Cited by 1,481
Digital pathology and artificial intelligence
Niazi, M. K. K., Parwani, A. V., & Gurcan, M. N. (2019) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30154-8 · Cited by 1,232
Deep Learning: A Primer for Radiologists
Chartrand, G., Cheng, P. M., Vorontsov, E., Drozdzal, M., Turcotte, S., Pal, C. J., Kadoury, S., & Tang, A. (2017) · RadioGraphics · doi:10.1148/rg.2017170077 · Cited by 1,115
Deep learning in histopathology: the path to the clinic
van der Laak, J., Litjens, G., & Ciompi, F. (2021) · Nature Medicine · doi:10.1038/s41591-021-01343-4 · Cited by 902
Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis
Karimi, D., Dou, H., Warfield, S. K., & Gholipour, A. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101759 · Cited by 706
Deep learning in medical imaging and radiation therapy
Sahiner, B., Pezeshk, A., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., Summers, R. M., & Giger, M. L. (2019) · Medical Physics · doi:10.1002/mp.13264 · Cited by 740
Deep neural network models for computational histopathology: A survey
Srinidhi, C. L., Ciga, O., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101813 · Cited by 749
A review on deep learning in medical image analysis
Suganyadevi, S., Seethalakshmi, V., & Balasamy, K. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-021-00218-1 · Cited by 653
“Random projection is a more computationally effective dimensionality reduction approach than principal component analysis [45].”
MethodTellez, D., Litjens, G., Bándi, P., Bulten, W., Bokhorst, J.-M., Ciompi, F., & van der Laak, J. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101544 · Cited by 649
Deep Learning and Its Applications in Biomedicine
Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., & Xie, Z. (2018) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2017.07.003 · Cited by 624
“On the other hand, crowdsourcing approaches have begun to pave the way in collecting annotations [191,192], which may be an important tool in the next few years.”
Background3D deeply supervised network for automated segmentation of volumetric medical images
Dou, Q., Yu, L., Chen, H., Jin, Y., Yang, X., Qin, J., & Heng, P.-A. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.05.001 · Cited by 613
Deep learning for image-based cancer detection and diagnosis − A survey
Hu, Z., Tang, J., Wang, Z., Zhang, K., Zhang, L., & Sun, Q. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.05.014 · Cited by 509
Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey
Bhattacharya, S., Reddy Maddikunta, P. K., Pham, Q.-V., Gadekallu, T. R., Krishnan S, S. R., Chowdhary, C. L., Alazab, M., & Jalil Piran, M. (2021) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102589 · Cited by 483
“Object detection is a key technology behind applications such as video surveillance, image retrieval system, and medical diagnostics (Albarqouni et al., 2016).”
Yu, H., Yang, L. T., Zhang, Q., Armstrong, D., & Deen, M. J. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.04.157 · Cited by 412
Deep learning in microscopy image analysis: a survey
Xing, F., Xie, Y., Su, H., Liu, F., & Yang, L. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2017.2766168 · Cited by 460
Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review
Bernal, J., Kushibar, K., Asfaw, D. S., Valverde, S., Oliver, A., Martí, R., & Lladó, X. (2019) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.08.008 · Cited by 368
“widely used in brain MRI for preprocessing data [15], detecting and segmenting lesions [16, 17, 18, 19, 20, 21] and segmenting tumours [22, 23, 24], whole tissue [25, 26, 27] and sub-cortical structures [28, 29].”
MethodThorax disease classification with attention guided convolutional neural network
Guan, Q., Huang, Y., Zhong, Z., Zheng, Z., Zheng, L., & Yang, Y. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.11.040 · Cited by 471
Cancer Diagnosis Using Deep Learning: A Bibliographic Review
Munir, K., Elahi, H., Ayub, A., Frezza, F., & Rizzi, A. (2019) · Cancers · doi:10.3390/cancers11091235 · Cited by 422
“One of the multi-scale CNN architectures, as described by researchers in [48], consists of three convolutional blocks, each of which comprises a convolutional layer and a rectified linear unit (ReLU) followed by the max-pooling layer and two fully connected layers.”
Surgical data science – from concepts toward clinical translation
Maier‐Hein, L., Eisenmann, M., Sarıkaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feußner, H., Giannarou, S., Mascagni, P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102306 · Cited by 512
Self-citationDeep and machine learning techniques for medical imaging-based breast cancer: A comprehensive review
Houssein, E. H., Emam, M. M., Ali, A. A., & Suganthan, P. N. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114161 · Cited by 404
Salvi, M., Acharya, U. R., Molinari, F., & Meiburger, K. M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104129 · Cited by 386
“Finally, Albarqouni et al. [163] implemented a multi-scale CNN for mitosis detection, combining the output probability of three different CCNs.”
MethodStructural crack detection using deep convolutional neural networks
Ali, R., Chuah, J. H., Talip, M. S. A., Mokhtar, N., & Shoaib, M. A. (2022) · Automation in Construction · doi:10.1016/j.autcon.2021.103989 · Cited by 369
Korotcov, A., Tkachenko, V., Russo, D. P., & Ekins, S. (2017) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.7b00578 · Cited by 352
“Previously deep learning has been used mostly for unsupervised learning and noisy data 28–34 .”
MethodDeep Convolutional Neural Networks for Chest Diseases Detection
Abiyev, R. H., & Ma’aitah, M. K. S. (2018) · Journal of Healthcare Engineering · doi:10.1155/2018/4168538 · Cited by 359
“Deep learning-based systems have been applied to increase the accuracy of image classification [10, 11].”
MethodGraham, S., Chen, H., Gamper, J., Dou, Q., Heng, P.-A., Snead, D., Tsang, Y. W., & Rajpoot, N. (2019) · Medical Image Analysis · doi:10.1016/j.media.2018.12.001 · Cited by 338
Pneumonia detection in chest X-ray images using convolutional neural networks and transfer learning
Jain, R., Nagrath, P., Kataria, G., Sirish Kaushik, V., & Jude Hemanth, D. (2020) · Measurement · doi:10.1016/j.measurement.2020.108046 · Cited by 325
Deep Learning Assisted Efficient AdaBoost Algorithm for Breast Cancer Detection and Early Diagnosis
Zheng, J., Lin, D., Gao, Z., Wang, S., He, M., & Fan, J. (2020) · IEEE Access · doi:10.1109/access.2020.2993536 · Cited by 315
“Deep learning is an emerging field where machine learning and AI use multiple nonlinear processing layers to learn features straightly from the information [15].”
BackgroundCervical cancer classification using convolutional neural networks and extreme learning machines
Ghoneim, A., Muhammad, G., & Hossain, M. S. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2019.09.015 · Cited by 312
The impact of site-specific digital histology signatures on deep learning model accuracy and bias
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24698-1 · Cited by 330
Celik, Y., Talo, M., Yildirim, O., Karabatak, M., & Acharya, U. R. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.03.011 · Cited by 318
Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge
Veta, M., Heng, Y. J., Stathonikos, N., Bejnordi, B. E., Beca, F., Wollmann, T., Rohr, K., Shah, M. A., Wang, D., Rousson, M., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.02.012 · Cited by 315
Pellegrini, E., Ballerini, L., Hernandez, M. D. C. V., Chappell, F. M., González‐Castro, V., Anblagan, D., Danso, S., Muñoz‐Maniega, S., Job, D., Pernet, C., et al. (2018) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2018.07.004 · Cited by 292
A comparative review on multi-modal sensors fusion based on deep learning
Tang, Q., Liang, J., & Zhu, F. (2023) · Signal Processing · doi:10.1016/j.sigpro.2023.109165 · Cited by 280
Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks
Matek, C., Schwarz, S., Spiekermann, K., & Marr, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0101-9 · Cited by 281
Deep Learning for Plant Diseases: Detection and Saliency Map Visualisation
Brahimi, M., Arsenovic, M., Laraba, S., Sladojevic, S., Boukhalfa, K., & Moussaoui, A. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_6 · Cited by 259
Deep learning in mammography and breast histology, an overview and future trends
Hamidinekoo, A., Denton, E., Rampun, A., Honnor, K., & Zwiggelaar, R. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.03.006 · Cited by 273
Data Privacy in Healthcare: In the Era of Artificial Intelligence
Yadav, N., Pandey, S., Gupta, A., Dudani, P., Gupta, S., & Rangarajan, K. (2023) · Indian Dermatology Online Journal · doi:10.4103/idoj.idoj_543_23 · Cited by 210
Machine Learning‐Enabled Smart Sensor Systems
Ha, N., Xu, K., Ren, G., Mitchell, A., & Ou, J. Z. (2020) · Advanced Intelligent Systems · doi:10.1002/aisy.202000063 · Cited by 207
Matek, C., Krappe, S., Münzenmayer, C., Haferlach, T., & Marr, C. (2021) · Blood · doi:10.1182/blood.2020010568 · Cited by 191
Lekadir, K., Galimzianova, A., Betriu, A., del Mar Vila, M., Igual, L., Rubin, D. L., Fernandez, E., Radeva, P., & Napel, S. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2631401 · Cited by 212
“In the last year, the paradigm has been applied extensively in several medical imaging applications, such as for brain [27], [28], lung [29], [30], and breast [31], [32] imaging.”
BackgroundRise of the Machines: Advances in Deep Learning for Cancer Diagnosis
Levine, A. B., Schlosser, C., Grewal, J., Coope, R., Jones, S. J. M., & Yip, S. (2019) · Trends in Cancer · doi:10.1016/j.trecan.2019.02.002 · Cited by 201
Who Said What: Modeling Individual Labelers Improves Classification
Guan, M., Gulshan, V., Dai, A., & Hinton, G. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11756 · Cited by 191
Learning from Disagreement: A Survey
Uma, A. N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., & Poesio, M. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12752 · Cited by 144
Efficient deep learning model for mitosis detection using breast histopathology images
Saha, M., Chakraborty, C., & Racoceanu, D. (2018) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2017.12.001 · Cited by 194
“…Learning based [42] Deep Learning Convolutional neural network (CNN) [43] Deep CNN and max-pooling [10] CNN and Handcrafted Features [17] CNN and seeded blob features [22] AggNet: Deep learning from crowd [44] H.Wang et al ., [47], [48] used CNN and handcrafted features for mitotic figure detection.”
MethodClassification of dental diseases using CNN and transfer learning
Prajapati, S. A., Nagaraj, R., & Mitra, S. (2017) · 2017 5th International Symposium on Computational and Business Intelligence (ISCBI) · doi:10.1109/iscbi.2017.8053547 · Cited by 179
A machine learning-based framework for diagnosis of COVID-19 from chest X-ray images
Rasheed, J., Hameed, A. A., Djeddi, C., Jamil, A., & Al-Turjman, F. (2021) · Interdisciplinary Sciences: Computational Life Sciences · doi:10.1007/s12539-020-00403-6 · Cited by 187
Learning laparoscopic video shot classification for gynecological surgery
Petscharnig, S., & Schöffmann, K. (2018) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4699-5 · Cited by 182
RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification
Wang, S., Zhu, Y., Yu, L., Chen, H., Lin, H., Wan, X., Fan, X., & Heng, P.-A. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101549 · Cited by 178
Zhang, J., Saha, A., Zhu, Z., & Mazurowski, M. A. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2865671 · Cited by 175
“[53] proposed an additional crowdsourcing layer (AggNet) to handle the data aggregation issue for mitosis detection in breast cancer histology images.”
BackgroundDeep learning in digital pathology image analysis: a survey
Deng, S., Zhang, X., Yan, W., Chang, E. I.-C., Fan, Y., Lai, M., & Xu, Y. (2020) · Frontiers of Medicine · doi:10.1007/s11684-020-0782-9 · Cited by 160
Weakly supervised mitosis detection in breast histopathology images using concentric loss
Li, C., Wang, X., Liu, W., Latecki, L. J., Wang, B., & Huang, J. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.013 · Cited by 160
Machine Learning Algorithm Validation
Maleki, F., Muthukrishnan, N., Ovens, K., Reinhold, C., & Forghani, R. (2020) · Neuroimaging Clinics of North America · doi:10.1016/j.nic.2020.08.004 · Cited by 140
Crowdsourcing in medical research: concepts and applications
Tucker, J. D., Day, S., Tang, W., & Bayus, B. (2019) · PeerJ · doi:10.7717/peerj.6762 · Cited by 158
Karimi, D., Nir, G., Fazli, L., Black, P. C., Goldenberg, L., & Salcudean, S. E. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2944643 · Cited by 181
“However, there exist more elaborate techniques [62]–[64].”
Classification of Histopathological Biopsy Images Using Ensemble of Deep Learning NetworksPreprint
Kassani, S. H., Kassani, P. H., Wesolowski, M. J., Schneider, K. A., & Deters, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11870 · Cited by 196
“For this experiment, Macenko et al. [22] approach is applied due to its promising performance in many studies [4, 32, 35, 42] to standardize the color intensity of the tissue.”
MethodMedical image fusion based on convolutional neural networks and non-subsampled contourlet transform
Wang, Z., Li, X., Duan, H., Su, Y., Zhang, X., & Guan, X. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114574 · Cited by 144
Association of Pathological Fibrosis With Renal Survival Using Deep Neural Networks
Kolachalama, V. B., Singh, P., Lin, C. Q., Mun, D., Belghasem, M. E., Henderson, J. M., Francis, J. M., Salant, D. J., & Chitalia, V. C. (2018) · Kidney International Reports · doi:10.1016/j.ekir.2017.11.002 · Cited by 137
“31 9,13,15 – 18 ,20 ,21 ,23 ,25 ,28 To evaluate the performance of the CNN model on the same biopsy samples, we performed a computer-based analysis of the digitized biopsy images and related the fi ndings to the outcomes of interest.”
MethodAutomated Analysis for Retinopathy of Prematurity by Deep Neural Networks
Hu, J., Chen, Y., Zhong, J., Ju, R., & Zhang, Y. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2863562 · Cited by 148
“Numerous studies have explored deep neural networks in a range of medical image analysis applications, including mitosis detection [34], lymph node detection [35], lung pattern classification [36], and breast cancer classification [37], etc.”
BackgroundEnsemble Knowledge Transfer for Semantic Segmentation
Nigam, I., Huang, C., & Ramanan, D. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00168 · Cited by 117
“imaging [45, 10, 1], and satellite imaging [23, 22, 33].”
Mapping of Crowdsourcing in Health: Systematic Review
Créquit, P., Mansouri, G., Benchoufi, M., Vivot, A., & Ravaud, P. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9330 · Cited by 135
“selected for further evaluation, and 202 studies were included (182 published papers and 20 conference abstracts [3,16-216]).”
BackgroundProstate cancer classification with multiparametric MRI transfer learning model
Yuan, Y., Qin, W., Buyyounouski, M., Ibragimov, B., Hancock, S., Han, B., & Xing, L. (2019) · Medical Physics · doi:10.1002/mp.13367 · Cited by 133
Madani, M., Behzadi, M. M., & Nabavi, S. (2022) · Cancers · doi:10.3390/cancers14215334 · Cited by 147
“In the other work performed in [269] for nuclei detection, a series of handcrafted features and features extracted from CNN were combined for better detection.”
MethodIntroduction to digital pathology and computer-aided pathology
Nam, S., Chong, Y., Jung, C. K., Kwak, T.-Y., Lee, J. Y., Park, J., Rho, M. J., & Go, H. (2020) · Journal of Pathology and Translational Medicine · doi:10.4132/jptm.2019.12.31 · Cited by 125
“Crowdsourcing has been used successfully for several goals [44,45].”
MethodLe, H., Gupta, R., Hou, L., Abousamra, S., Fassler, D., Torre-Healy, L., Moffitt, R. A., Kurc, T., Samaras, D., Batiste, R., et al. (2020) · The American Journal of Pathology · doi:10.1016/j.ajpath.2020.03.012 · Cited by 124
Breast Cancer Histopathological Image Classification: A Deep Learning ApproachPreprint
Motlagh, M. H., Jannesari, M., Es, H. A., Khosravi, P., Elemento, O., Totonchi, M., & Hajirasouliha, I. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/242818 · Cited by 141
“Moreover, AggNet system which is a combination of CNN and additional crowd-sourcing layer successfully detected mitosis in breast cancer images [15].”
Background3D convolutional neural network for automatic detection of lung nodules in chest CT
Hamidian, S., Sahiner, B., Petrick, N., & Pezeshk, A. X. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2255795 · Cited by 127
GoogLeNet-Based Ensemble FCNet Classifier for Focal Liver Lesion Diagnosis
Balagourouchetty, L., Jayanthi, K., Pottakkat, B., & Ramkumar, G. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2942774 · Cited by 138
“A diverse set of such pretrained networks namely Alexnet [15], GoogLeNet [16], VGGNet [17] & [18], Cifar-Net [19], AggNet [20] and so on are developed to compete successfully in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).”
BackgroundChoi, J. W., Ku, Y., Yoo, B. W., Kim, J. A., Lee, D. S., Chai, Y. J., Kong, H., & Kim, H. C. (2017) · PLoS ONE · doi:10.1371/journal.pone.0189259 · Cited by 129
Segmentation of the hippocampus by transferring algorithmic knowledge for large cohort processing
Thyreau, B., Sato, K., Fukuda, H., & Taki, Y. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.11.004 · Cited by 123
3-D Convolutional Neural Networks for Automatic Detection of Pulmonary Nodules in Chest CT
Pezeshk, A., Hamidian, S., Petrick, N., & Sahiner, B. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2879449 · Cited by 112
“visual tasks for 2D natural and medical images [4]–[6].”
BackgroundRashmi, R., Prasad, K., & Udupa, C. B. K. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01786-9 · Cited by 130
“Many studies have been carried out on the classification of mitotic and non-mitotic nuclei using DL techniques [163–166].”
MethodMotepe, S., Hasan, A. N., & Stopforth, R. (2019) · IEEE Access · doi:10.1109/access.2019.2923796 · Cited by 119
Breast Cancer Histopathological Image Classification: A Deep Learning Approach
Jannesari, M., Habibzadeh, M., Es, H. A., Khosravi, P., Elemento, O., Totonchi, M., & Hajirasouliha, I. (2018) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621307 · Cited by 111
A Comprehensive Survey on Deep-Learning-Based Breast Cancer Diagnosis
Mridha, M. F., Hamid, M. A., Monowar, M. M., Keya, A. J., Ohi, A. Q., Islam, M. R., & Kim, J.-M. (2021) · Cancers · doi:10.3390/cancers13236116 · Cited by 114
Deep neural network for automatic characterization of lesions on 68Ga-PSMA-11 PET/CT
Zhao, Y. S., Gafita, A., Vollnberg, B., Tetteh, G., Haupt, F., Afshar‐Oromieh, A., Menze, B. H., Eiber, M. J., Rominger, A., & Shi, K. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04606-y · Cited by 107
General framework, opportunities and challenges for crowdsourcing techniques: A Comprehensive survey
Bhatti, S. S., Gao, X., & Chen, G. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110611 · Cited by 122
Deep computational pathology in breast cancer
Duggento, A., Conti, A., Mauriello, A., Guerrisi, M., & Toschi, N. (2020) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.08.006 · Cited by 106
“Authors of [95] attempt to understand whether the labels generated from non-expert users (which inevitably lead to noisy annotations) can be employed in a massive scale to train a deep CNN.”
BackgroundCheplygina, V., de Bruijne, M., & Pluim, J. P. W. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.03.009 · Cited by 93
Machine Learning Methods for Histopathological Image Analysis: A Review
de Matos, J., Ataky, S. T. M., Britto, A. D. S., Oliveira, L. S., & Koerich, A. L. (2021) · Electronics · doi:10.3390/electronics10050562 · Cited by 109
Capsule Networks Against Medical Imaging Data Challenges
Jiménez-Sánchez, A., Albarqouni, S., & Mateus, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01364-6_17 · Cited by 90
“Although strategies like transfer learning [14], data augmentation [12] or crowdsourcing [2] have been proposed, data collection and annotations is for many medical applications still a bottleneck [3].”
BackgroundSelf-citationSiamese Networks for Chromosome Classification
Swati, Gupta, G., Yadav, M., Sharma, M., & Vig, L. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.17 · Cited by 106
“While we utilize crowdsourcing for a small portion of our task of chromosome classification, crowdsourcing has been actively contributing in many ways to generate necessary annotations for various similar tasks such as image segmentation [29], mitosis detection [3] and nucleus detection [13].”
MethodMachine learning in computational histopathology: Challenges and opportunities
Cooper, M., Ji, Z., & Krishnan, R. G. (2023) · Genes, Chromosomes and Cancer · doi:10.1002/gcc.23177 · Cited by 79
Miccio, L., Cimmino, F., Kurelac, I., Villone, M. M., Bianco, V., Memmolo, P., Merola, F., Mugnano, M., Capasso, M., Iolascon, A., et al. (2020) · View · doi:10.1002/viw.20200034 · Cited by 91
Survey on Big Data Analytics in Health Care
S, S. P., & Asha, P. (2019) · International Conference on Smart Systems and Inventive Technology (ICSSIT) · doi:10.1109/icssit46314.2019.8987882 · Cited by 95
Liu, B., Chi, W., Li, X., Li, P., Liang, W., Liu, H., Wang, W., & He, J. (2020) · Journal of Cancer Research and Clinical Oncology · doi:10.1007/s00432-019-03098-5 · Cited by 62
Automatic classification of tissue malignancy for breast carcinoma diagnosis
Fondón, I., Sarmiento, A., García, A. I., Silvestre, M., Eloy, C., Polónia, A., & Aguiar, P. (2018) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2018.03.003 · Cited by 87
Breast Cancer Detection From Histopathological Images Using Deep Learning
Khuriwal, N., & Mishra, N. (2018) · IEEE International Conference on Recent Advances and Innovations in Engineering (ICRAIE) · doi:10.1109/icraie.2018.8710426 · Cited by 82
Prospects of deep learning for medical imaging
Kim, J., Hong, J., & Park, H. (2018) · Precision and Future Medicine · doi:10.23838/pfm.2018.00030 · Cited by 89
Deep Convolutional Neural Network for Chicken Diseases Detection
Mbelwa, H., Mbelwa, J., & Machuve, D. Z. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120295 · Cited by 81
“They have been used for diagnosis and detection in various fields including medical, agricultural and other fields [9], [10].”
MethodA Survey of Crowdsourcing in Medical Image Analysis
Ørting, S. N., Doyle, A., Van Hilten, A., Hirth, M., Inel, O., Madan, C. R., Mavridis, P., Spiers, H., & Cheplygina, V. (2020) · Human Computation · doi:10.15346/hc.v7i1.111 · Cited by 48
Moscalu, M., Moscalu, R., Dascălu, C. G., Ţarcă, V., Cojocaru, E., Costin, I. M., Țarcă, E., & Șerban, I. L. (2023) · Diagnostics · doi:10.3390/diagnostics13142379 · Cited by 93
“…specimen; due to the high volume of processed information and the algorithms complexity, however, research tends to focus on narrower objectives that require the analysis of smaller images areas, such as mitosis detection, anatomical region identification or tumor process identification [1,58,59].”
BackgroundCheng, J., Mo, X., Wang, X., Parwani, A., Feng, Q., & Huang, K. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx723 · Cited by 79
“Plenty of methods have been proposed for tasks like object detection and segmentation (nuclei, mitosis, and glands) (Albarqouni et al. , 2016; Xu et al. , 2014), benign/malignant image classification (Zhang et al. , 2015a), and discovery of prognostic factors (Beck et al. , 2011; Chen et al. ,…”
MethodActive learning with point supervision for cost-effective panicle detection in cereal crops
Chandra, A. L., Desai, S. V., Balasubramanian, V. N., Ninomiya, S., & Guo, W. (2020) · Plant Methods · doi:10.1186/s13007-020-00575-8 · Cited by 80
“Crowd-sourcing [15–21] has been effectively used to generate datasets to train deep learning models for visual tasks.”
MethodA Unified Framework for Tumor Proliferation Score Prediction in Breast Histopathology
Paeng, K., Hwang, S., Park, S., & Kim, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_27 · Cited by 81
“In order to solve the problem of subjectivity in the mitosis counting task, automatic mitosis detection algorithms have been proposed in recent years [1,4,17] that perform mitosis detection within regions pre-selected by pathologists.”
Method
Metadata
- Authors
- Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N.
- Affiliations
- German Center for Neurodegenerative Diseases (DE)
- Technical University of Munich (DE)
- University of Oxford (GB)
- Johns Hopkins University (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1313-1321
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2339885376
- Cited by
- 606
- References
- 38
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 59 · 2023: 59 · 2022: 60 · 2021: 92 · 2020: 89 · 2019: 80 · 2018: 73 · 2017: 46 · 2016: 10
- FWCI
- 57.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- DZNE Pub (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- mitosis detection, breast cancer histopathology, crowdsourced annotation, noisy annotations, data aggregation, convolutional neural networks, CrowdFlower
- MeSH terms
- Machine Learning, Breast Neoplasms, Female, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Mitosis, Neural Networks, Computer, Video Games, Internet, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.