Constrained-CNN losses for weakly supervised segmentation
Open accessThe DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
&NA; Weakly‐supervised learning based on, e.g., partially labelled images or image‐tags, is currently attracting significant attention in CNN segmentation as it can mitigate the need for full and laborious pixel/voxel annotations. Enforcing high‐order (global) inequality constraints on the network output (for instance, to constrain the size of the target region) can leverage unlabeled data, guiding the training process with domain‐specific knowledge. Inequality constraints are very flexible because they do not assume exact prior knowledge. However, constrained Lagrangian dual optimization has been largely avoided in deep networks, mainly for computational tractability reasons. To the best of our knowledge, the method of Pathak et al. (2015a) is the only prior work that addresses deep CNNs with linear constraints in weakly supervised segmentation. It uses the constraints to synthesize fully‐labeled training masks (proposals) from weak labels, mimicking full supervision and facilitating dual optimization. We propose to introduce a differentiable penalty, which enforces inequality constraints directly in the loss function, avoiding expensive Lagrangian dual iterates and proposal generation. From constrained‐optimization perspective, our simple penalty‐based approach is not optimal as there is no guarantee that the constraints are satisfied. However, surprisingly, it yields substantially better results than the Lagrangian‐based constrained CNNs in Pathak et al. (2015a), while reducing the computational demand for training. By annotating only a small fraction of the pixels, the proposed approach can reach a level of segmentation performance that is comparable to full supervision on three separate tasks. While our experiments focused on basic linear constraints such as the target‐region size and image tags, our framework can be easily extended to other non‐linear constraints, e.g., invariant shape moments (Klodt and Cremers, 2011) and other region statistics (Lim et al., 2014). Therefore, it has the potential to close the gap between weakly and fully supervised learning in semantic medical image segmentation. Our code is publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
(2016) · Cited by 11,679
(2017) · Cited by 11,625
SLIC Superpixels Compared to State-of-the-Art Superpixel Methods
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunk, S. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.120 · Cited by 9,201
GrabCut: Interactive foreground extraction using iterated graph cuts
Rother, C., Kolmogorov, V., & Blake, A. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015720 · Cited by 5,703
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2015) · doi:10.1109/cvpr31182.2015 · Cited by 3,737
2017 IEEE International Conference on Computer Vision (ICCV)
(2017) · doi:10.1109/iccv37128.2017 · Cited by 3,056
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · cited 2×
“For all the experiments, we used ENet [15], as it has shown a good trade-off between accuracy and inference time.”
Method2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2016) · doi:10.1109/cvpr33180.2016 · Cited by 1,534
2015 IEEE International Conference on Computer Vision (ICCV)
(2015) · doi:10.1109/iccv33071.2015 · Cited by 1,253
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation
Lin, D., Dai, J., Jia, J., He, K., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.344 · Cited by 1,129 · cited 2×
“In fact, most of the existing techniques synthesize fully-labeled training masks (proposals) from the available partial labels, mimicking full supervision [19, 14, 9, 6].”
BackgroundBoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
Dai, J., He, K., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.191 · Cited by 1,092
Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation
Papandreou, G., Chen, L.-C., Murphy, K. P., & Yuille, A. L. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.203 · Cited by 1,005
What’s the Point: Semantic Segmentation with Point Supervision
Bearman, A., Russakovsky, O., Ferrari, V., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_34 · Cited by 887
Neural Information Processing Systems
Anderson, D. Z. (1988) · neural information processing systems · Cited by 784
Simple Does It: Weakly Supervised Instance and Semantic Segmentation
Khoreva, A., Benenson, R., Hosang, J., Hein, M., & Schiele, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.181 · Cited by 764
Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation
Kolesnikov, A., & Lampert, C. H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_42 · Cited by 719
From image-level to pixel-level labeling with Convolutional Networks
Pinheiro, P. O., & Collobert, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298780 · Cited by 673
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation
Pathak, D., Krahenbuhl, P., & Darrell, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.209 · Cited by 617
STC: A Simple to Complex Framework for Weakly-Supervised Semantic Segmentation
Wei, Y., Liang, X., Chen, Y., Shen, X., Cheng, M.-M., Feng, J., Zhao, Y., & Yan, S. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2636150 · Cited by 576
Train faster, generalize better: stability of stochastic gradient descent
Hardt, M., Recht, B., & Singer, Y. (2016) · International Conference on Machine Learning · Cited by 541
Deep Learning via Semi-supervised Embedding
Weston, J., Ratle, F., Mobahi, H., & Collobert, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_34 · Cited by 534
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Zhang, Y., David, P., & Gong, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.223 · Cited by 494
Computer Vision (ICCV), 2011 IEEE International Conference on
Vezhnevets, A. S., Ferrari, V., & Buhmann, J. M. (2011) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 477
Semi-supervised Learning for Network-Based Cardiac MR Image Segmentation
Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P. M., & Rueckert, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66185-8_29 · Cited by 416
3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study
Dolz, J., Desrosiers, C., & Ben Ayed, I. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.039 · Cited by 414
Self-citationDeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks
Rajchl, M., Lee, M. C. H., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M. A., Hajnal, J. V., Kainz, B., et al. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2621185 · Cited by 410
Lagrange programming neural networks
Zhang, S., & Constantinides, A. G. (1992) · IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing · doi:10.1109/82.160169 · Cited by 399
FusionNet: A Deep Fully Residual Convolutional Neural Network for Image Segmentation in Connectomics
Quan, T. M., Hildebrand, D. G. C., & Jeong, W.-K. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.613981 · Cited by 279
On Regularized Losses for Weakly-supervised CNN Segmentation
Tang, M., Perazzi, F., Djelouah, A., Ayed, I. B., Schroers, C., & Boykov, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_31 · Cited by 247
Self-citationConstrained deep weak supervision for histopathology image segmentation
Jia, Z., Huang, X., Chang, E. I.-C., & Xu, Y. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2724070 · Cited by 242
Fully Convolutional Multi-Class Multiple Instance Learning
Pathak, D., Shelhamer, E., Long, J., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.7144 · Cited by 236
Semi-Supervised 3D Abdominal Multi-Organ Segmentation Via Deep Multi-Planar Co-Training
Zhou, Y., Wang, Y., Tang, P., Bai, S., Shen, W., Fishman, E. K., & Yuille, A. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00020 · Cited by 211
Region Classification with Markov Field Aspect Models
Verbeek, J., & Triggs, B. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383098 · Cited by 189
Learning to segment under various forms of weak supervision
Xu, J., Schwing, A. G., & Urtasun, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299002 · Cited by 180
Weakly supervised semantic segmentation with a multi-image model
Vezhnevets, A. S., Ferrari, V., & Buhmann, J. M. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126299 · Cited by 163
Towards weakly supervised semantic segmentation by means of multiple instance and multitask learning
Vezhnevets, A., & Buhmann, J. M. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540060 · Cited by 161
Arnab, A., Zheng, S., Jayasumana, S., Romera‐Paredes, B., Larsson, M., Kirillov, A., Savchynskyy, B., Rother, C., Kahl, F., & Torr, P. H. S. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2762355 · Cited by 158
Weakly supervised structured output learning for semantic segmentation
Vezhnevets, A. S., Ferrari, V., & Buhmann, J. M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247757 · Cited by 157
Tell Me What You See and I Will Show You Where It Is
Xu, J. W., Schwing, A. G., & Urtasun, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.408 · Cited by 103
Imposing Hard Constraints on Deep Networks: Promises and LimitationsPreprint
Márquez-Neila, P., Salzmann, M., & Fua, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02025 · Cited by 91
Normalized Cut Loss for Weakly-supervised CNN SegmentationPreprint
Tang, M., Djelouah, A., Perazzi, F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01346 · Cited by 53
Self-citationFast Trust Region for Segmentation
Gorelick, L., Schmidt, F. R., & Boykov, Y. Y. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.224 · Cited by 47
Self-citationA convex framework for image segmentation with moment constraints
Klodt, M., & Cremers, D. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126502 · Cited by 44
Learning under Distributed Weak Supervision
Rajchl, M., Lee, M. C. H., Schrans, F., Davidson, A., Passerat‐Palmbach, J., Tarroni, G., Alansary, A., Oktay, O., Kainz, B., & Rueckert, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01100 · Cited by 29
Segmentation with area constraints
Niethammer, M., & Zach, C. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.09.002 · Cited by 18
Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-TrainingPreprint
Zhou, Y., Wang, Y., Tang, P., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02586 · Cited by 17
Constrained Deep Learning using Conditional Gradient and Applications in Computer VisionPreprint
Ravi, S. N., Dinh, T., Lokhande, V. S. R., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06453 · Cited by 11
Efficient Energy Minimization for Enforcing Label Statistics
Lim, Y., Jung, K., & Kohli, P. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2306415 · Cited by 10
Volumetric Bias in Segmentation and Reconstruction: Secrets and Solutions
Boykov, Y., Isack, H. N., Olsson, C., et al. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.206 · Cited by 9
Self-citation
Cited by
Zhou, S. K., Greenspan, H., Davatzikos, C., Duncan, J. S., Van Ginneken, B., Madabhushi, A., Prince, J. L., Rueckert, D., & Summers, R. M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3054390 · Cited by 1,118
Deep semantic segmentation of natural and medical images: a review
Asgari Taghanaki, S., Abhishek, K., Cohen, J. P., Cohen-Adad, J., & Hamarneh, G. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09854-1 · Cited by 870
Deep Learning for Cardiac Image Segmentation: A Review
Chen, C., Qin, C., Qiu, H., Tarroni, G., Duan, J., Bai, W., & Rueckert, D. (2020) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2020.00025 · Cited by 869
AI applications to medical images: From machine learning to deep learning
Castiglioni, I., Rundo, L., Codari, M., Di Leo, G., Salvatore, C., Interlenghi, M., Gallivanone, F., Cozzi, A., D'Amico, N. C., & Sardanelli, F. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.02.006 · Cited by 710
Qureshi, I., Yan, J., Abbas, Q., Shaheed, K., Riaz, A. B., Wahid, A., Khan, M. W. J., & Szczuko, P. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.031 · Cited by 295
Metadata
- Authors
- Kervadec, H., Dolz, J., Tang, M., Granger, E., Boykov, Y., & Ben Ayed, I.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2019-05
- Volume
- 54
- Pages
- 88-99
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2799738340
- Cited by
- 278
- References
- 57
- FWCI
- 17.54
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.