The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Structured Knowledge Distillation for Dense Prediction
Open accessAbstract
In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at https://git.io/StructKD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
FCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013
Self-citationImproved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · cited 2×
“The Lipschitz constraint is enforced by applying a gradient penalty [59].”
BackgroundMethodSqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Deep High-Resolution Representation Learning for Human Pose Estimation
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00584 · Cited by 5,984
Self-citationContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894
Self-citationSemantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
CornerNet: Detecting Objects as Paired Keypoints
Law, H., & Deng, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_45 · Cited by 3,852
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Unified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · cited 5×
“73% improvement over the one without distillation reported in [66].”
BackgroundMethodInfluentialSelf-citationDeep Ordinal Regression Network for Monocular Depth Estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00214 · Cited by 1,991
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709
DenseASPP for Semantic Segmentation in Street Scenes
Yang, M., Yu, K., Zhang, C., Li, Z., & Yang, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00388 · Cited by 1,674
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation
Lin, G., Shen, C., van den Hengel, A., & Reid, I. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.348 · Cited by 862
Self-citationSemantic Segmentation using Adversarial Networks
Luc, P., Couprie, C., Chintala, S., & Verbeek, J. (2016) · arXiv · doi:10.48550/arxiv.1611.08408 · Cited by 611
Cited by
FCOS: A Simple and Strong Anchor-free Object Detector
Tian, Z., Shen, C., Chen, H., & He, T. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3032166 · Cited by 890
Self-citationConditional Convolutions for Instance Segmentation
Tian, Z., Shen, C., & Chen, H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_17 · Cited by 712
Self-citationChannel-wise Knowledge Distillation for Dense Prediction
Shu, C., Liu, Y., Gao, J., Yan, Z., & Shen, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00526 · Cited by 452
Self-citationCross-Image Relational Knowledge Distillation for Semantic Segmentation
Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., & Zhang, Q. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01200 · Cited by 265
Reconstructed Graph Neural Network With Knowledge Distillation for Lightweight Anomaly Detection
Zhou, X., Wu, J., Liang, W., Wang, K. I.-K., Yan, Z., Yang, L. T., & Jin, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3389714 · Cited by 145
Metadata
- Authors
- Liu, Y., Shu, C., Wang, J., & Shen, C.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-06-01
- Volume
- 45
- Issue
- 6
- Pages
- 7035-7049
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3034870355
- Cited by
- 197
- References
- 34
- FWCI
- 11.68
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.