A survey on active learning and human-in-the-loop deep learning for medical image analysis
Open accessAbstract
Fully automatic deep learning has become the state-of-the-art technique for many tasks including image acquisition, analysis and interpretation, and for the extraction of clinically useful information for computer-aided detection, diagnosis, treatment planning, intervention and therapy. However, the unique challenges posed by medical image analysis suggest that retaining a human end-user in any deep learning enabled system will be beneficial. In this review we investigate the role that humans might play in the development and deployment of deep learning enabled diagnostic applications and focus on techniques that will retain a significant input from a human end user. Human-in-the-Loop computing is an area that we see as increasingly important in future research due to the safety-critical nature of working in the medical domain. We evaluate four key areas that we consider vital for deep learning in the clinical practice: (1) Active Learning to choose the best data to annotate for optimal model performance; (2) Interaction with model outputs - using iterative feedback to steer models to optima for a given prediction and offering meaningful ways to interpret and respond to predictions; (3) Practical considerations - developing full scale applications and the key considerations that need to be made before deployment; (4) Future Prospective and Unanswered Questions - knowledge gaps and related research fields that will benefit human-in-the-loop computing as they evolve. We offer our opinions on the most promising directions of research and how various aspects of each area might be unified towards common goals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous reviews are more oriented to other AI areas such as natural language processing, speech processing, and computer vision, without going deeper into OD, showing only a few related papers.
“Furthermore, previous reviews [8,10-13] are more oriented to other AI areas such as natural language processing, speech processing, and computer vision, but without going deeper into OD, showing only a few related papers.”
From Ten Years of Active Learning Techniques and Object Detection: A Systematic Review (2023) · cites this paper as [11]
The introduction (p. 2) identifies this as a review of human-in-the-loop deep learning for medical image analysis and says it focuses on medical image segmentation, supporting the point that it is not an in-depth review of object detection (OD). Our copy does not establish that it shows only a few OD-related papers. The references to natural language and speech processing appear to describe the cited reviews collectively, not necessarily this source. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThere are three main active learning scenarios: membership query synthesis, stream-based selective sampling, and pool-based sampling.
“Generally, there are three main scenarios in AL, as discussed in [11,35]:”
From Ten Years of Active Learning Techniques and Object Detection: A Systematic Review (2023) · cites this paper as [11]
Five recent active learning reviews were identified in the preliminary search.
“Five recent AL reviews [8,10-13] were identified in that process”
From Ten Years of Active Learning Techniques and Object Detection: A Systematic Review (2023) · cites this paper as [11]
New standards are currently being developed in industry to guide businesses on how to define, implement and measure data quality throughout the ML development lifecycle.
“Similar trends are happening in industry, where new standards are currently being developed to guide businesses on how to define, implement and measure data quality throughout the ML development lifecycle [18].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [18]
Section 4, “Practical Considerations” (p. 11), discusses developing and deploying deep-learning applications, while Section 5 (p. 15) calls for baseline methods to assess their performance. Neither passage addresses industry standards for defining, implementing, or measuring data quality across the ML development lifecycle; searches for “industry standards” and “data quality” found no passage that does. (LLM, conf=0.76)
Experimental — accuracy not yet measuredActive learning methods could enable selection of uncertain samples or users for further analysis such as additional review by domain experts (physicians).
“Furthermore, active learning methods could enable the selection of uncertain samples or users for further analysis such as an additional review by domain experts (physicians) [186].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [186]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486 · see citing works
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1993.341010 · Cited by 14,474 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Attention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458 · see citing works
Self-citationDeep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968 · see citing works
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682 · cited 1× · see citing works
“We refer the reader to a number of reviews of DL in medical imaging (Hesamian et al., 2019; Lundervold and Lundervold, 2019; Yamashita et al., 2018).”
A theory of learning from different domains
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010) · Machine Learning · doi:10.1007/s10994-009-5152-4 · Cited by 3,515 · see citing works
Mathematical Theory of Communication
Dubitzky, W., Olaf Wolkenhauer, A., Cho, K.-H., & Yokota, H. (2013) · Springer eBooks · doi:10.1007/978-1-4419-9863-7_100015 · Cited by 3,291 · see citing works
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · see citing works
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117 · see citing works
Advances in Neural Information Processing Systems 37
Touretzky, D. S. (2024) · doi:10.52202/079017 · Cited by 2,874 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · see citing works
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · see citing works
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058 · see citing works
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014 · see citing works
Self-citationAngluin, D. (1988) · Machine Learning · doi:10.1023/a:1022821128753 · Cited by 1,918 · see citing works
Generative adversarial network in medical imaging: A review
Xin, Y., Walia, E., & Babyn, P. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101552 · Cited by 1,883 · see citing works
Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges
Hesamian, M. H., Jia, W., He, X., & Kennedy, P. J. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00227-x · Cited by 1,644 · see citing works
Improving Generalization with Active Learning
Cohn, D. A., Atlas, L., & Ladner, R. E. (1994) · Machine Learning · doi:10.1023/a:1022673506211 · Cited by 1,597 · see citing works
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571 · see citing works
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240 · see citing works
Heterogeneous Uncertainty Sampling for Supervised Learning
Lewis, D. D., & Catlett, J. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50026-x · Cited by 1,174 · see citing works
Overview of deep learning in medical imaging
Suzuki, K. (2017) · Radiological Physics and Technology · doi:10.1007/s12194-017-0406-5 · Cited by 1,102 · see citing works
Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation
Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J. N., Wu, Z., & Ding, X. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101693 · Cited by 1,095 · see citing works
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2018) · NORA - Norwegian Open Research Archives · Cited by 1,012 · see citing works
An analysis of active learning strategies for sequence labeling tasks
Settles, B., & Craven, M. W. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613855 · Cited by 991 · see citing works
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Braud, C., Lacroix, O., & Søgaard, A. (2017) · doi:10.18653/v1/d17-1 · Cited by 910 · see citing works
Deep Bayesian Active Learning with Image Data
Gal, Y., Islam, R., & Ghahramani, Z. (2017) · arXiv · doi:10.48550/arxiv.1703.02910 · Cited by 900 · see citing works
Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(1997) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1997.609286 · Cited by 853 · see citing works
Transfusion: Understanding Transfer Learning for Medical ImagingPreprint
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07208 · Cited by 845 · see citing works
Interactive medical image segmentation using deep learning with image-specific fine tuning
Wang, G., Li, W., Zuluaga, M. A., Pratt, R., Patel, P. A., Aertsen, M., Doel, T., David, A. L., Deprest, J., Ourselin, S., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2791721 · Cited by 843 · see citing works
BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active LearningPreprint
Kirsch, A., van Amersfoort, J. R., & Gal, Y. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1906.08158 · Cited by 826 · see citing works
Deep learning enables cross-modality super-resolution in fluorescence microscopy
Wang, H., Rivenson, Y., Jin, Y., Wei, Z., Gao, R., Günaydın, H., Bentolila, L. A., Kural, C., & Özcan, A. (2018) · Nature Methods · doi:10.1038/s41592-018-0239-0 · Cited by 788 · see citing works
Employing EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689 · see citing works
King, R. D., Rowland, J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., et al. (2009) · Science · doi:10.1126/science.1165620 · Cited by 679 · see citing works
Cost-Effective Active Learning for Deep Image Classification
Wang, K., Zhang, D., Li, Y., Zhang, R., & Lin, L. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2016.2589879 · Cited by 669 · see citing works
Functional genomic hypothesis generation and experimentation by a robot scientist
King, R. D., Whelan, K. E., Jones, F. M., Reiser, P. G. K., Bryant, C. H., Muggleton, S. H., Kell, D. B., & Oliver, S. G. (2004) · Nature · doi:10.1038/nature02236 · Cited by 662 · see citing works
Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., & Vercauteren, T. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.103 · Cited by 659 · see citing works
Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1
Zoubin Ghahramani, A., Welling, M., Cortes, C., Lawrence, N. D., & Weinberger, K. Q. (2014) · Cited by 656 · see citing works
The Power of Ensembles for Active Learning in Image Classification
Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00976 · Cited by 655 · see citing works
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · see citing works
Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing
Lapata, M., & Ng, H. T. (2008) · Cited by 555 · see citing works
Artificial Intelligence and Digital Pathology: Challenges and Opportunities
Tizhoosh, H. R., & Pantanowitz, L. (2018) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_53_18 · Cited by 544 · see citing works
Lian, C., Liu, M., Zhang, J., & Shen, D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889096 · Cited by 544 · see citing works
Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation
Yang, L., Zhang, Y., Chen, J., Zhang, S., & Chen, D. Z. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66179-7_46 · Cited by 526 · see citing works
Unsupervised Domain Adaptation in Brain Lesion Segmentation with Adversarial Networks
Kamnitsas, K., Baumgartner, C. F., Ledig, C., Newcombe, V., Simpson, J., Kane, A. D., Menon, D. K., Nori, A., Criminisi, A., Rueckert, D., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-59050-9_47 · Cited by 525 · see citing works
A scoping review of transfer learning research on medical image analysis using ImageNet
Morid, M. A., Borjali, A., & Del Fiol, G. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104115 · Cited by 508 · see citing works
Committee-Based Sampling For Training Probabilistic Classifiers
Dagan, I., & Engelson, S. P. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50027-x · Cited by 506 · see citing works
DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
Wang, G., Zuluaga, M. A., Li, W., Pratt, R., Patel, P. A., Aertsen, M., Doel, T., David, A. L., Deprest, J., Ourselin, S., et al. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2840695 · Cited by 502 · see citing works
Data Augmentation Using Learned Transformations for One-Shot Medical Image Segmentation
Zhao, A. T., Balakrishnan, G., Durand, F., Guttag, J. V., & Dalca, A. V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00874 · Cited by 476 · see citing works
Interpretability of deep learning models: A survey of results
Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., Srivastava, M. B., Preece, A. D., Julier, S., Rao, R., et al. (2017) · IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) · doi:10.1109/uic-atc.2017.8397411 · Cited by 469 · see citing works
Proceedings of the Fifteenth International Conference on Machine Learning
SHAVLIK, J. W. (1998) · Cited by 460 · see citing works
A Survey on Shari’Ah Governance Practices in Malaysia, GCC Countries and the UKPreprint
Hasan, Z. (2011) · SSRN Electronic Journal · Cited by 452 · see citing works
3D conditional generative adversarial networks for high-quality PET image estimation at low dose
Wang, Y., Yu, B., Wang, L., Zu, C., Lalush, D. S., Lin, W., Wu, X., Zhou, J., Shen, D. G., & Zhou, L. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.03.045 · Cited by 452 · see citing works
Space–time wiring specificity supports direction selectivity in the retina
EyeWirers, T., Kim, J. S., Greene, M. J., Zlateski, A., Lee, K., Richardson, M., Turaga, S. C., Purcaro, M., Balkam, M., Robinson, A., et al. (2014) · Nature · doi:10.1038/nature13240 · Cited by 450 · see citing works
Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally
Zhou, Z., Shin, J., Zhang, L., Gurudu, S., Gotway, M. B., & Liang, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.506 · Cited by 430 · see citing works
Landmark-based deep multi-instance learning for brain disease diagnosis
Liu, M., Zhang, J., Adeli, E., & Shen, D. G. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.10.005 · Cited by 424 · see citing works
DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks
Rajchl, M., Lee, M. C. H., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M. A., Hajnal, J. V., Kainz, B., et al. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2621185 · Cited by 410 · see citing works
Self-citationDeep Learning for Multi-task Medical Image Segmentation in Multiple Modalities
Moeskops, P., Wolterink, J. M., van der Velden, B. H. M., Gilhuijs, K. G. A., Leiner, T., Viergever, M. A., & Išgum, I. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_55 · Cited by 354 · see citing works
Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image SegmentationPreprint
Tajbakhsh, N., Jeyaseelan, L., Li, Q., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10454 · Cited by 324 · see citing works
Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image Synthesis
Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., & Bourgeat, P. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2895894 · Cited by 301 · see citing works
GeoS: Geodesic Image Segmentation
Criminisi, A., Sharp, T., & Blake, A. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_9 · Cited by 277 · see citing works
Learning how to Active Learn: A Deep Reinforcement Learning Approach
Fang, M., Li, Y., & Cohn, T. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1063 · Cited by 262 · see citing works
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244 · see citing works
Interactive Image Segmentation via Backpropagating Refinement Scheme
Jang, W.-D., & Kim, C.-S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00544 · Cited by 234 · see citing works
Training Connectionist Networks with Queries and Selective Sampling
Atlas, L., Cohn, D. A., & Ladner, R. E. (1989) · neural information processing systems · Cited by 220 · see citing works
An active learning framework for content-based information retrieval
Zhang, C., & Chen, T. (2002) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2002.1017738 · Cited by 205 · see citing works
Mahapatra, D., Bozorgtabar, B., Thiran, J., & Reyes, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00934-2_65 · Cited by 182 · see citing works
3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training
Xia, Y., Liu, F., Yang, D., Cai, J., Yu, L., Zhu, Z., Xu, D., Yuille, A., & Roth, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093608 · Cited by 170 · see citing works
An investigation of player motivations in Eyewire, a gamified citizen science project
Tinati, R., Luczak–Roesch, M., Simperl, E. P. B., & Hall, W. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2016.12.074 · Cited by 166 · see citing works
Learning Algorithms for Active LearningPreprint
Bachman, P., Sordoni, A., & Trischler, A. (2017) · International Conference on Machine Learning · doi:10.48550/arxiv.1708.00088 · Cited by 166 · see citing works
Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation
Jungo, A., & Reyes, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_6 · Cited by 156 · see citing works
2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
da Costa, V. G. T., Zara, G., Rota, P., Oliveira-Santos, T., Sebe, N., Murino, V., & Elisa, R. (2021) · doi:10.1109/wacv48630.2021 · Cited by 131 · see citing works
Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital PathologyPreprint
Holzinger, A., Malle, B., Kieseberg, P., Roth, P., Müller, H., Reihs, R., & Zatloukal, K. (2017) · arXiv.org · doi:10.48550/arxiv.1712.06657 · Cited by 113 · see citing works
Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., & Zhang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00941 · Cited by 113 · see citing works
Hauptmann, A. G., Lin, W., Yan, R., et al. (2006) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1180639.1180721 · Cited by 109 · see citing works
On Statistical Bias In Active Learning: How and When To Fix ItPreprint
Farquhar, S., Gal, Y., & Rainforth, T. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2101.11665 · Cited by 106 · see citing works
Cost-Sensitive Active Learning for Intracranial Hemorrhage Detection
Kuo, W., Häne, C., Yuh, E. L., Mukherjee, P., & Malik, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00931-1_82 · Cited by 104 · see citing works
Practical Obstacles to Deploying Active Learning
Lowell, D., Lipton, Z. C., & Wallace, B. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1003 · Cited by 92 · see citing works
Active One-shot LearningPreprint
Woodward, M. P., & Finn, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.06559 · Cited by 89 · see citing works
Pan, Y., Liu, M.-X., Lian, C., Xia, Y., & Shen, D. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2983085 · Cited by 87 · see citing works
Understanding and Interpreting Machine Learning in Medical Image Computing Applications
Pereira, S., Meier, R., Alves, V., Reyes, M., & Silva, C. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02628-8 · Cited by 85 · see citing works
Advancing medical imaging informatics by deep learning-based domain adaptation
Choudhary, A., Tong, L., Zhu, Y., & Wang, M. D. (2020) · Yearbook of Medical Informatics · doi:10.1055/s-0040-1702009 · Cited by 84 · cited 1× · see citing works
“We refer the reader to Guan and Liu (2021); Choudhary et al. (2020) for an overview of DA methods used for medical image analysis, and hope to see greater application of DA methods in AL scenarios in the future.”
MethodEye Tracking for Deep Learning Segmentation Using Convolutional Neural Networks
Stember, J. N., Çelik, H., Krupinski, E. A., Chang, P., Mutasa, S., Wood, B. J., Lignelli, A., Moonis, G., Schwartz, L. H., Jambawalikar, S., et al. (2019) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-019-00220-4 · Cited by 69 · see citing works
Active, continual fine tuning of convolutional neural networks for reducing annotation efforts
Zhou, Z., Shin, J. Y., Gurudu, S., Gotway, M. B., & Liang, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101997 · Cited by 64 · see citing works
Human-Machine Collaboration for Medical Image Segmentation
Ravanbakhsh, M., Tschernezki, V., Last, F., Klein, T., Batmanghelich, K. N., Tresp, V., & Nabi, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053555 · Cited by 60 · see citing works
Active Learning from Weak and Strong LabelersPreprint
Zhang, C., & Chaudhuri, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.02847 · Cited by 55 · see citing works
Kushibar, K., Valverde, S., González-Villà, S., Bernal, J., Cabezas, M., Oliver, A., & Lladó, X. (2019) · Scientific Reports · doi:10.1038/s41598-019-43299-z · Cited by 50 · see citing works
Multi-Label Learning from Crowds
Li, S.-Y., Jiang, Y., Chawla, N. V., et al. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2857766 · Cited by 49 · see citing works
Angluin, D. (2001) · Discovery Science · Cited by 47 · see citing works
Wen, S., Kurç, T., Hou, L., Saltz, J. H., Gupta, R., Batiste, R. C., Zhao, T., Nguyen, V., Samaras, D., & Zhu, W. (2018) · Europe PMC (PubMed Central) · Cited by 47 · see citing works
Towards continual learning in medical imagingPreprint
Baweja, C., Glocker, B., & Kamnitsas, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02496 · Cited by 45 · see citing works
Machine Learning Proceedings 1994
(1994) · Elsevier eBooks · doi:10.1016/c2009-0-27542-8 · Cited by 40 · see citing works
Artificial intelligence for melanoma diagnosis: how can we deliver on the promise?
Mar, V. J., & Soyer, H. P. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy193 · Cited by 37 · cited 1× · see citing works
“Developments in the field of computer vision have shown great promise in transferring to medical image analysis, and several techniques have been shown to perform as accurately as human observers (Haenssle et al., 2018; Mar and Soyer, 2018).”
BackgroundConfident Head Circumference Measurement from Ultrasound with Real-Time Feedback for Sonographers
Budd, S., Sinclair, M. D., Khanal, B., Matthew, J., Lloyd, D. F. A., Gómez, A., Toussaint, N., Robinson, E. C., & Kainz, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_75 · Cited by 36 · see citing works
Self-citationActive Learning for Segmentation by Optimizing Content Information for Maximal Entropy
Özdemir, F., Peng, Z., Tanner, C., Fürnstahl, P., & Göksel, O. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00889-5_21 · Cited by 35 · see citing works
Iterative Interaction Training for Segmentation Editing Networks
Bredell, G., Tanner, C., & Konukoğlu, E. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00919-9_42 · Cited by 32 · see citing works
Learning under Distributed Weak SupervisionPreprint
Rajchl, M., Lee, M. C. H., Schrans, F., Davidson, A., Passerat‐Palmbach, J., Tarroni, G., Alansary, A., Oktay, O., Kainz, B., & Rueckert, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01100 · Cited by 30 · see citing works
Self-citationZhou, Z., Shin, J., Feng, R., Hurst, R. T., Kendall, C. B., & Liang, J. (2018) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-018-0143-2 · Cited by 30 · see citing works
Straight to the Point: Reinforcement Learning for User Guidance in Ultrasound
Milletarì, F., Birodkar, V., & Sofka, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32875-7_1 · Cited by 30 · see citing works
Active Deep Learning with Fisher Information for Patch-Wise Semantic Segmentation
Sourati, J., Gholipour, A., Dy, J., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00889-5_10 · Cited by 29 · see citing works
Geometry in active learning for binary and multi-class image segmentation
Konyushkova, K., Sznitman, R., & Fua, P. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.01.007 · Cited by 25 · see citing works
Mixed-Supervised Dual-Network for Medical Image Segmentation
Wang, D., Li, M., Ben‐Shlomo, N., Corrales, C. E., Cheng, Y., Zhang, T., & Jayender, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32245-8_22 · Cited by 25 · see citing works
Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation
Khosravan, N., Çelik, H., Türkbey, B., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61188-4_9 · Cited by 18 · see citing works
MedAL: Deep Active Learning Sampling Method for Medical Image AnalysisPreprint
Smailagic, A., Noh, H. Y., Costa, P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.09287 · Cited by 16 · see citing works
Bhalgat, Y., Shah, M., & Awate, S. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11302 · Cited by 16 · see citing works
An interactive learning framework for scalable classification of pathology images
Nalisnik, M., Gutman, D. A., Kong, J., et al. (2015) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2015.7363841 · Cited by 15 · see citing works
Learning Active Learning from Real and Synthetic Data
Konyushkova, K., Sznitman, R., & Fua, P. (2017) · arXiv.org · doi:10.48550/arxiv.1703.03365 · Cited by 14 · see citing works
Employing Weak Annotations for Medical Image Analysis ProblemsPreprint
Rajchl, M., Koch, L. M., Ledig, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06297 · Cited by 10 · see citing works
A Semi-Automated Usability Evaluation Framework for Interactive Image Segmentation Systems
Amrehn, M., Steidl, S., Kortekaas, R. W. L., Strumia, M., Weingarten, M., Kowarschik, M., & Maier, A. (2019) · International Journal of Biomedical Imaging · doi:10.1155/2019/1464592 · Cited by 10 · see citing works
Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image SegmentationPreprint
Khosravan, N., Çelik, H., Türkbey, B., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.03235 · Cited by 7 · see citing works
Enslaving the Algorithm: From a Right to an Explanationn to a Right to Better Decisionss?Preprint
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3052831 · Cited by 5 · see citing works
Kurzendorfer, T., Fischer, P., Mirshahzadeh, N., et al. (2017) · Journal of Imaging · doi:10.3390/jimaging3040056 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Amrehn, M., Gaube, S., Unberath, M., Schebesch, F., Horz, T., Strumia, M., Steidl, S., Kowarschik, M., Maier, A., 2017. UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model, in: Bruckner, S., Hennemuth, A., Kainz, B., Hotz, I., Merhof, D., Rieder, C. (Eds.), Eurographics Workshop on Visual Computing for Biology and Medicine, The Eurographics Association. doi: 10 : 2312/vcbm : 20171248 .
- Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A., de Bruijne, M., 2016. Early experiences with crowdsourcing airway annotations in chest CT, in: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer Verlag. pp. 209-218. URL: https://link : springer : com/chapter/ 10 : 1007/978-3-319-46976-8 22 , doi: 10 : 1007/978-3-319-469768{ n }22 .
- Guan, H., Liu, M., 2021. Domain Adaptation for Medical Image Analysis: A Survey URL: http://arxiv : org/abs/2102 : 09508 .
- Rodrigues, F., Pereira, F.C., 2018. Deep learning from crowds, in: The ThirtySecond AAAI Conference on Artificial Intelligence (AAAI), 2018, AAAI Press. pp. 1611-1618.
- Zhou, Z., Shin, J., Feng, R., Hurst, R.T., Kendall, C.B., Liang, J., 2018b. Integrating Active Learning and Transfer Learning for Carotid Intima-Media Thickness Video Interpretation. Journal of Digital Imaging URL: http://www : ncbi : nlm : nih : gov/pubmed/30402668http: //link : springer : com/10 : 1007/s10278-018-0143-2 , doi: 10 : 1007/ s10278-018-0143-2 .
Cited by
Sorted by citations
Cellpose 2.0: how to train your own model
Pachitariu, M., & Stringer, C. (2022) · Nature Methods · doi:10.1038/s41592-022-01663-4 · Cited by 1,381 · see citing works
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169 · see citing works
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800 · see citing works
Challenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638 · see citing works
Deep learning for Alzheimer's disease diagnosis: A survey
Khojaste-Sarakhsi, M., Haghighi, S. S., Ghomi, S. M. T. F., & Marchiori, E. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102332 · Cited by 194 · see citing works
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90 · see citing works
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86 · see citing works
A multifaceted vision of the human-AI collaboration: a comprehensive review
Puerta-Beldarrain, M., Gómez-Carmona, O., Sánchez-Corcuera, R., Casado-Mansilla, D., López-de-Ipiña, D., & Chen, L. (2025) · IEEE Access · doi:10.1109/access.2025.3536095 · Cited by 73 · see citing works
Felmingham, C. M., Adler, N. R., Ge, Z., Morton, R. L., Janda, M., & Mar, V. J. (2021) · American Journal of Clinical Dermatology · doi:10.1007/s40257-020-00574-4 · Cited by 57 · see citing works
Constructing Ethical AI Based on the “Human-in-the-Loop” System
Chen, X., Wang, X., & Qu, Y. (2023) · Systems · doi:10.3390/systems11110548 · Cited by 45 · see citing works
Khosravi, P., Fuchs, T. J., & Ho, D. J. (2025) · Cancer Research · doi:10.1158/0008-5472.can-24-3630 · Cited by 41 · see citing works
Human-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30 · see citing works
Aresta, G., Araújo, T., Fazekas, B., Mai, J., Schmidt-Erfurth, U., & Bogunović, H. (2024) · IEEE Access · doi:10.1109/access.2024.3379015 · Cited by 5 · see citing works
“An alternative is to use networks that account for possible user inputs without the need for additional weight refinement [23], [28], [29], [30], [31], [32].”
Method
- Authors
- Budd, S., Robinson, E. C., & Kainz, B.
- Affiliations
- Imperial College London (GB)
- King's College London (GB)
- Corresponding author
- Budd, Samuel
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2021-07
- Volume
- 71
- Pages
- 102062
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2977942577
- Cited by
- 606
- References
- 121
- Citations by year
- 2026: 84 · 2025: 110 · 2024: 138 · 2023: 127 · 2022: 99 · 2021: 42 · 2020: 6
- FWCI
- 8.47
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Medical Image Analysis (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Imperial College London
- King's College London EPSRC Centre for Doctoral Training in Medical Imaging
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- active learning (machine learning), clinical annotation, model interpretability, iterative feedback, model deployment, safety-critical systems, computer-aided diagnosis, human-machine interaction
- MeSH terms
- Deep Learning, Humans, Image Processing, Computer-Assisted
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
