A deep learning-based interactive medical image segmentation framework with sequential memory
Open accessAbstract
Background and objective Image segmentation is an essential component in medical image analysis. The case of 3D images such as MRI is particularly challenging and time consuming. Interactive or semi-automatic methods are thus highly desirable. However, existing methods do not exploit the typical sequentiality of real user interactions. This is due to the interaction memory used in these systems, which discards ordering. In contrast, we argue that the order of the user corrections should be used for training and lead to performance improvements. Methods We contribute to solving this problem by proposing a general multi-class deep learning-based interactive framework for image segmentation, which embeds a base network in a user interaction loop with a user feedback memory. We propose to model the memory explicitly as a sequence of consecutive system states, from which the features can be learned, generally learning from the segmentation refinement process. Training is a major difficulty owing to the network's input being dependent on the previous output. We adapt the network to this loop by introducing a virtual user in the training process, modelled by dynamically simulating the iterative user feedback. Results We evaluated our framework against existing methods on the complex task of multi-class semantic instance female pelvis MRI segmentation with 5 classes, including up to 27 tumour instances, using a segmentation dataset collected in our hospital, and on liver and pancreas CT segmentation, using public datasets. We conducted a user evaluation, involving both senior and junior medical personnel in matching and adjacent areas of expertise. We observed an annotation time reduction with 5'56" for our framework against 25' on average for classical tools. We systematically evaluated the influence of the number of clicks on the segmentation accuracy. A single interaction round our framework outperforms existing automatic systems with a comparable setup. We provide an ablation study and show that our framework outperforms existing interactive systems. Conclusions Our framework largely outperforms existing systems in accuracy, with the largest impact on the smallest, most difficult classes, and drastically reduces the average user segmentation time with fast inference at 47.2±6.2 ms per image.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054 · see citing works
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,281 · see citing works
N4ITK: Improved N3 Bias Correction
Tustison, N. J., Avants, B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., & Gee, J. C. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2046908 · Cited by 6,119 · see citing works
2008 IEEE Conference on Computer Vision and Pattern Recognition
(2008) · doi:10.1109/cvpr12618.2008 · Cited by 4,263 · see citing works
Image segmentation using deep learning: A survey
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059968 · Cited by 3,796 · see citing works
Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images
Boykov, Y. Y., & Jolly, M.-P. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937505 · Cited by 3,674 · see citing works
Brain tumor segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.05.004 · Cited by 3,278 · see citing works
2017 IEEE International Conference on Computer Vision (ICCV)
(2017) · doi:10.1109/iccv37128.2017 · Cited by 3,129 · see citing works
Random Walks for Image Segmentation
Grady, L. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.233 · Cited by 2,624 · see citing works
Self-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393 · see citing works
Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015
Navab, N., Joachim Hornegger, A., Wells, W. M., & Frangi, A. F. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9 · Cited by 2,288 · see citing works
NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection
Ghiasi, G., Lin, T.-Y., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00720 · Cited by 2,099 · see citing works
U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications
Siddique, N. A., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021) · IEEE Access · doi:10.1109/access.2021.3086020 · Cited by 2,060 · see citing works
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
(2020) · doi:10.1109/cvpr42600.2020 · Cited by 1,805 · see citing works
The Medical Segmentation Decathlon
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B. A., Litjens, G., Menze, B., Ronneberger, O., Summers, R. M., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30695-9 · Cited by 1,384 · see citing works
3D Slicer: A Platform for Subject-Specific Image Analysis, Visualization, and Clinical Support
Kikinis, R., Pieper, S., & Vosburgh, K. G. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7657-3_19 · Cited by 1,175 · see citing works
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
(2019) · doi:10.1109/iccv43118.2019 · Cited by 1,136 · see citing works
Deep Learning vs. Traditional Computer Vision
O’Mahony, N., Campbell, S., Carvalho, A. L. D., Harapanahalli, S., Velasco-Hernandez, G., Krpálková, L., Riordan, D., & Walsh, J. L. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17795-9_10 · Cited by 1,104 · see citing works
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018
Frangi, A. F., Schnabel, J. A., Davatzikos, C. A., Alberola-López, C., & Fichtinger, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00937-3 · Cited by 1,049 · see citing works
Rother, C., Kolmogorov, V., & Blake, A. (2004) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1186562.1015720 · Cited by 935 · see citing works
Interactive medical image segmentation using deep learning with image-specific fine tuning
Wang, G., Li, W., Zuluaga, M. A., Pratt, R., Patel, P. A., Aertsen, M., Doel, T., David, A. L., Deprest, J., Ourselin, S., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2791721 · Cited by 843 · see citing works
Expectation-Maximization Attention Networks for Semantic Segmentation
Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., & Liu, H. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00926 · Cited by 741 · see citing works
DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation
Wang, G., Zuluaga, M. A., Li, W., Pratt, R., Patel, P. A., Aertsen, M., Doel, T., David, A. L., Deprest, J., Ourselin, S., et al. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2840695 · Cited by 502 · see citing works
Lung Infection Quantification of COVID-19 in CT Images with Deep LearningPreprint
Shan, F., Gao, Y., Wang, J., Shi, W., Shi, N., Han, M., Xue, Z., Shen, D. G., & Shi, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04655 · Cited by 489 · see citing works
The Medical Imaging Interaction Toolkit
Ivo Wolf, A., Vetter, M., Wegner, I., Böttger, T., Nolden, M., Schöbinger, M., Hastenteufel, M., Kunert, T., & Meinzer, H.-P. (2005) · Medical Image Analysis · doi:10.1016/j.media.2005.04.005 · Cited by 424 · see citing works
Cardenas, C. E., Yang, J., Anderson, B., Court, L. E., & Brock, K. (2019) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2019.02.001 · Cited by 411 · see citing works
GeoS: Geodesic Image Segmentation
Criminisi, A., Sharp, T., & Blake, A. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_9 · Cited by 277 · see citing works
Reviving Iterative Training with Mask Guidance for Interactive Segmentation
Sofiiuk, K., Petrov, I. A., & Konushin, A. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897365 · Cited by 264 · see citing works
Deformable MR Prostate Segmentation via Deep Feature Learning and Sparse Patch Matching
Guo, Y., Gao, Y., & Shen, D. G. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2508280 · Cited by 254 · see citing works
F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation
Sofiiuk, K., Petrov, I. A., Barinova, O., & Konushin, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00865 · Cited by 239 · see citing works
Interactive Image Segmentation via Backpropagating Refinement Scheme
Jang, W.-D., & Kim, C.-S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00544 · Cited by 234 · see citing works
Zhu, H., Meng, F., Cai, J., & Lu, S. (2015) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2015.10.012 · Cited by 233 · see citing works
FocalClick: Towards Practical Interactive Image Segmentation
Chen, X., Zhao, Z., Zhang, Y., Duan, M., Qi, D., & Zhao, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00136 · Cited by 182 · see citing works
Kline, T. L., Korfiatis, P. D., Edwards, M. E., Blais, J. D., Czerwiec, F. S., Harris, P. C., King, B. F., Torres, V. E., & Erickson, B. J. (2017) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-017-9978-1 · Cited by 151 · see citing works
NuClick: A deep learning framework for interactive segmentation of microscopic images
Alemi Koohbanani, N., Jahanifar, M., Zamani Tajadin, N., & Rajpoot, N. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101771 · Cited by 147 · see citing works
Johnson, J. W. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-17798-0 · Cited by 113 · see citing works
Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Wang, Y., & Zhang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00941 · Cited by 113 · see citing works
Volumetric memory network for interactive medical image segmentation
Zhou, T., Li, L., Bredell, G., Li, J., Unkelbach, J., & Konukoglu, E. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102599 · Cited by 109 · see citing works
Augmented Reality Guided Laparoscopic Surgery of the Uterus
Collins, T., Pizarro, D., Gasparini, S., Bourdel, N., Chauvet, P., Canis, M. J., Calvet, L., & Bartoli, A. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3027442 · Cited by 107 · see citing works
Self-citation2021 IEEE International Conference on Image Processing (ICIP)
(2021) · doi:10.1109/icip42928.2021 · Cited by 91 · see citing works
Angiodysplasia Detection and Localization Using Deep Convolutional Neural Networks
Shvets, A. A., Iglovikov, V., Rakhlin, A., & Kalinin, A. A. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00098 · Cited by 82 · see citing works
A survey of recent interactive image segmentation methods
Ramadan, H., Lachqar, C., & Tairi, H. (2020) · Computational Visual Media · doi:10.1007/s41095-020-0177-5 · Cited by 71 · see citing works
FocusCut: Diving into a Focus View in Interactive Segmentation
Zheng, L. L., Duan, Z.-P., Zhang, Z., Guo, C., & Cheng, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00266 · Cited by 70 · see citing works
Arrastia, J. L., Heilenkötter, N., Baguer, D. O., Hauberg‐Lotte, L., Boskamp, T., Hetzer, S., Duschner, N., Schaller, J., & Maaß, P. (2021) · Journal of Imaging · doi:10.3390/jimaging7040071 · Cited by 66 · see citing works
Robust Interactive Semantic Segmentation of Pathology Images with Minimal User Input
Jahanifar, M., Tajeddin, N. Z., Koohbanani, N. A., & Rajpoot, N. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00081 · Cited by 26 · see citing works
Interactive Deep Editing Framework for Medical Image Segmentation
Zhou, B., Chen, L., & Zhao, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32248-9_37 · Cited by 16 · see citing works
Amrehn, M., Gaube, S., Unberath, M., et al. (2017) · arXiv (Cornell University) · doi:10.2312/vcbm.20171248 · Cited by 12 · see citing works
External Validation of Deep Learning Algorithms for Cardiothoracic Ratio Measurement
Chaisangmongkon, W., Chamveha, I., Promwiset, T., et al. (2021) · IEEE Access · doi:10.1109/access.2021.3101253 · Cited by 10 · see citing works
NuClick: A Deep Learning Framework for Interactive Segmentation of Microscopy ImagesPreprint
Koohbanani, N. A., Jahanifar, M., Tajadin, N. Z., & Rajpoot, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14511 · Cited by 9 · see citing works
A Deep Learning-Based Interactive Medical Image Segmentation Framework
Mikhailov, I. A., Chauveau, B., Bourdel, N., & Bartoli, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17721-7_11 · Cited by 6 · see citing works
Self-citationAutomatic measurement of cardiothoracic ratio in chest x-ray images with ProGAN-generated dataset
Kusakunniran, W., Saiviroonporn, P., Siriapisith, T., et al. (2023) · Applied Computing and Informatics · doi:10.1108/aci-11-2022-0322 · Cited by 4 · see citing works
Applications of Medical Artificial Intelligence
(2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17721-7 · Cited by 2 · see citing works
Докія Гуменна. Щоденник. Зошит 23
Shvets, T. (2018) · Філологічний дискурс · doi:10.31475/fil.dys.2018.08.20
Flagged first, then by citations
Cited by
Sorted by citations
Aresta, G., Araújo, T., Fazekas, B., Mai, J., Schmidt-Erfurth, U., & Bogunović, H. (2024) · IEEE Access · doi:10.1109/access.2024.3379015 · Cited by 5 · see citing works
“Previously corrected regions can become partially or totally lost, increasing the overall time required to achieve a satisfactory result [35], [36].”
BackgroundTian, Z., He, Y., Sun, L., Li, Y., & Du, S. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128184 · Cited by 5 · see citing works
- Authors
- Mikhailov, I., Chauveau, B., Bourdel, N., & Bartoli, A.
- Affiliations
- Institut Pascal (FR)
- Corresponding author
- Mikhailov, Ivan
- Venue
- Computer Methods and Programs in Biomedicine
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 245
- Pages
- 108038
- ISSN
- 0169-2607 (ISSN-L)
- OpenAlex
- W4391069643
- Semantic Scholar
- c77e40d300983a1a47730fbd585992799783501a
- Cited by
- 15
- References
- 60
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 5
- FWCI
- 2.10
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (am, 329 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Computer Methods and Programs in Biomedicine (acceptedVersion)
- PubMed (publishedVersion)
- HAL (Le Centre pour la Communication Scientifique Directe) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Keywords
- interactive segmentation, semi-automatic segmentation, sequential memory, multi-class segmentation, segmentation refinement
- MeSH terms
- Deep Learning, Female, Humans, Image Processing, Computer-Assisted, Liver, Magnetic Resonance Imaging, Tomography, X-Ray Computed, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
