PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers
Open accessAbstract
Two-branch network architecture has shown its efficiency and effectiveness in real-time semantic segmentation tasks. However, direct fusion of high-resolution details and low-frequency context has the drawback of detailed features being easily overwhelmed by surrounding contextual information. This overshoot phenomenon limits the improvement of the segmentation accuracy of existing two-branch models. In this paper, we make a connection between Convolutional Neural Networks (CNN) and Proportional-Integral-Derivative (PID) controllers and reveal that a two-branch network is equivalent to a Proportional-Integral (PI) controller, which inherently suffers from similar overshoot issues. To alleviate this problem, we propose a novel three-branch network architecture: PIDNet, which contains three branches to parse detailed, context and boundary information, respectively, and employs boundary attention to guide the fusion of detailed and context branches. Our family of PIDNets achieve the best trade-off between inference speed and accuracy and their accuracy surpasses all the existing models with similar inference speed on the Cityscapes and CamVid datasets. Specifically, PIDNet-S achieves 78.6% mIOU with inference speed of 93.2 FPS on Cityscapes and 80.1% mIOU with speed of 153.7 FPS on CamVid.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Encoder-decoder architectures and Transformer-based structures are widely used to model dependencies between modalities.
“For example, encoder-decoder architectures and Transformer-based structures are widely used to model dependencies between modalities [12,13];”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [12]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · cited 1× · see citing works
“Inspired from the long-range dependency parsing ability of attention mechanism [47] for language machine, non-local operation [49] was introduced into computer vision and led to many accurate models [17, 23, 55].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982 · see citing works
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517 · see citing works
Dual Attention Network for Scene Segmentation
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00326 · Cited by 7,351 · see citing works
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894 · see citing works
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · see citing works
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1× · see citing works
“Starting from FCN [31], which achieved great improvement over traditional methods, deep convnets gradually dominated the semantic segmentation field and many representative models have been proposed [4, 6, 40, 48, 59, 60].”
MethodRefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968 · see citing works
Training Region-Based Object Detectors with Online Hard Example Mining
Shrivastava, A., Gupta, A., & Girshick, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.89 · Cited by 2,686 · see citing works
BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation
Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., & Sang, N. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01515-2 · Cited by 1,813 · see citing works
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709 · see citing works
Context Encoding for Semantic Segmentation
Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., & Agrawal, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00747 · Cited by 1,510 · see citing works
The Role of Context for Object Detection and Semantic Segmentation in the Wild
Mottaghi, R., Chen, X., Liu, X., Cho, N.-G., Lee, S.-W., Fidler, S., Urtasun, R., & Yuille, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.119 · Cited by 1,438 · see citing works
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
Semantic object classes in video: A high-definition ground truth database
Brostow, G., Fauqueur, J., & Cipolla, R. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.04.005 · Cited by 1,400 · see citing works
Object-Contextual Representations for Semantic Segmentation
Yuan, Y., Chen, X., & Wang, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_11 · Cited by 1,285 · see citing works
Rethinking BiSeNet For Real-time Semantic Segmentation
Fan, M., Lai, S., Huang, J., Wei, X., Chai, Z., Luo, J., & Wei, X. K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00959 · Cited by 913 · see citing works
Deep semantic segmentation of natural and medical images: a review
Taghanaki, S. A., Abhishek, K., Cohen, J., Cohen‐Adad, J., & Hamarneh, G. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09854-1 · Cited by 888 · see citing works
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
Takikawa, T., Acuna, D., Jampani, V., & Fidler, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00533 · Cited by 870 · see citing works
A review of deep learning methods for semantic segmentation of remote sensing imagery
Yuan, X., Shi, J., & Gu, L. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114417 · Cited by 780 · see citing works
DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation
Li, H., Xiong, P., Fan, H., & Sun, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00975 · Cited by 772 · see citing works
Semantic Flow for Fast and Accurate Scene Parsing
Li, X., You, A., Zhu, Z., Zhao, H., Yang, M., Yang, K., Tan, S., & Tong, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_45 · Cited by 471 · see citing works
Fast-SCNN: Fast Semantic Segmentation NetworkPreprint
Poudel, R. P. K., Liwicki, S., & Cipolla, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04502 · Cited by 464 · see citing works
Oršić, M., Krešo, I., Bevandić, P., & Šegvić, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01289 · Cited by 437 · see citing works
Context Contrasted Feature and Gated Multi-scale Aggregation for Scene Segmentation
Ding, H., Jiang, X., Shuai, B., Liu, A. Q., & Wang, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00254 · Cited by 403 · see citing works
Automatic Instrument Segmentation in Robot-Assisted Surgery using Deep Learning
Shvets, A. A., Rakhlin, A., Kalinin, A. A., & Iglovikov, V. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00100 · Cited by 402 · see citing works
Learning to Predict Crisp Boundaries
Deng, R., Shen, C., Liu, S., Wang, H., & Liu, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_35 · Cited by 299 · see citing works
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation
Nirkin, Y., Wolf, L., & Hassner, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00405 · Cited by 277 · see citing works
Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes
Hong, Y., Pan, H., Sun, W., & Jia, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06085 · Cited by 261 · see citing works
Ayala, H. V. H., & Coelho, L. D. S. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.02.027 · Cited by 222 · see citing works
Temporally Distributed Networks for Fast Video Semantic Segmentation
Hu, P., Caba, F., Wang, O., Lin, Z., Sclaroff, S., & Perazzi, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00884 · Cited by 219 · see citing works
ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-timePreprint
Poudel, R. P. K., Bonde, U., Liwicki, S., & Zach, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04554 · Cited by 215 · see citing works
A new PID controller design for automatic generation control of hydro power systems
Khodabakhshian, A., & Hooshmand, R. (2009) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2009.11.006 · Cited by 195 · see citing works
Partial Order Pruning: For Best Speed/Accuracy Trade-Off in Neural Architecture Search
Li, X. S., Zhou, Y., Pan, Z. Y., & Feng, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00936 · Cited by 190 · see citing works
FasterSeg: Searching for Faster Real-time Semantic SegmentationPreprint
Chen, W., Gong, X., Liu, X., Zhang, Q., Li, Y., & Wang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.10917 · Cited by 177 · see citing works
Genetic Algorithm Based PID Controller Tuning Approach for Continuous Stirred Tank Reactor
Jayachitra, A., & Vinodha, R. (2014) · Advances in Artificial Intelligence · doi:10.1155/2014/791230 · Cited by 172 · see citing works
PP-LiteSeg: A Superior Real-Time Semantic Segmentation ModelPreprint
Peng, J., Liu, Y., Tang, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.02681 · Cited by 132 · see citing works
A PID Controller Approach for Stochastic Optimization of Deep Networks
An, W., Wang, H., Sun, Q., Xu, J., Dai, Q., & Lei Zhang, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00889 · Cited by 127 · see citing works
Self-citationReal-Time Semantic Segmentation via Multiply Spatial Fusion Network
Si, H., Zhang, Z., & Lu, F. (2020) · British Machine Vision Conference (BMVC) · doi:10.5244/c.34.153 · Cited by 59 · see citing works
Towards Fast and Robust Real Image Denoising With Attentive Neural Network and PID Controller
Ma, R., Li, S., Zhang, B., & Li, Z. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3079697 · Cited by 52 · see citing works
A PID Controller Architecture Inspired Enhancement to the PSO Algorithm
Xu, J., & Bhattacharyya, S. P. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-98012-2_42 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Representative Methods Year Encoder-Decoder Framework PIDNet [12] 2023 MAMT [62] 2022 DDRNet [61] 2021 PointRend [59] 2020 HRNet [60] 2019 DeepLabv3+ [58] 2018”
- Authors
- Xu, J., Xiong, Z., & Bhattacharyya, S. P.
- Affiliations
- Texas A&M University (US)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 19529-19539
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076267
- Semantic Scholar
- 8bcf60e213831d70a67de7cc033f984e5a908ca6
- Cited by
- 817
- References
- 55
- Citations by year
- 2026: 192 · 2025: 338 · 2024: 217 · 2023: 68 · 2022: 2
- FWCI
- 87.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (ECCS-1923803,CCF-2007527)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Civil and Structural Engineering › Infrastructure Maintenance and Monitoring
- Keywords
- real-time semantic segmentation, PID controller, feature fusion, overshoot, boundary information, inference speed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
