End-to-End Instance Segmentation with Recurrent Attention
Open accessAbstract
While convolutional neural networks have gained impressive success recently in solving structured prediction problems such as semantic segmentation, it remains a challenge to differentiate individual object instances in the scene. Instance segmentation is very important in a variety of applications, such as autonomous driving, image captioning, and visual question answering. Techniques that combine large graphical models with low-level vision have been proposed to address this problem, however, we propose an end-to-end recurrent neural network (RNN) architecture with an attention mechanism to model a human-like counting process, and produce detailed instance segmentations. The network is jointly trained to sequentially produce regions of interest as well as a dominant object segmentation within each region. The proposed model achieves competitive results on the CVPPP [27], KITTI [12], and Cityscapes [8] datasets.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several object detectors used the bipartite matching loss.
“Several object detectors [9,25,35] used the bipartite matching loss.”
From End-to-End Object Detection with Transformers (2020) · cites this paper as [35]
Cites
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Self-citationConvolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · see citing works
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · see citing works
Hypercolumns for object segmentation and fine-grained localization
Hariharan, B., Arbelaez, P., Girshick, R., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298642 · Cited by 1,511 · see citing works
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.04412 · Cited by 1,273 · see citing works
Instance-Aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.343 · Cited by 1,251 · see citing works
Simultaneous Detection and Segmentation
Hariharan, B., Arbeláez, P., Girshick, R., & Malik, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_20 · Cited by 1,176 · see citing works
Learning To Count Objects in Images
Lempitsky, V., & Zisserman, A. (2010) · Cited by 1,063 · see citing works
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · Cited by 785 · see citing works
Learning to Refine Object Segments
Pinheiro, P. O., Lin, T.-Y., Collobert, R., & Dollár, P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_5 · Cited by 768 · see citing works
Learning to Segment Object CandidatesPreprint
Pinheiro, P. H. O., Collobert, R., & Dollár, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06204 · Cited by 710 · see citing works
Exploring Models and Data for Image Question AnsweringPreprint
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 618 · see citing works
Self-citationDeep Watershed Transform for Instance Segmentation
Bai, M., & Urtasun, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.305 · Cited by 563 · see citing works
End-to-End People Detection in Crowded Scenes
Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · see citing works
Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation
Ghiasi, G., & Fowlkes, C. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_32 · Cited by 502 · see citing works
Search-based structured prediction
Daumé, H., Langford, J. C., & Marcu, D. (2009) · Machine Learning · doi:10.1007/s10994-009-5106-x · Cited by 465 · see citing works
Instance-Sensitive Fully Convolutional Networks
Dai, J., He, K., Li, Y., Ren, S., & Sun, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_32 · Cited by 421 · see citing works
Class-Specific, Top-Down Segmentation
Borenstein, E., & Ullman, S. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47967-8_8 · Cited by 367 · see citing works
Recurrent Instance Segmentation
Romera‐Paredes, B., & Torr, P. H. S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_19 · Cited by 324 · see citing works
Finely-grained annotated datasets for image-based plant phenotyping
Minervini, M., Fischbach, A., Scharr, H., & Tsaftaris, S. A. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2015.10.013 · Cited by 315 · see citing works
Leaf segmentation in plant phenotyping: a collation study
Scharr, H., Minervini, M., French, A. P., Klukas, C., Kramer, D., Liu, X., Luengo, I., Pape, J.-M., Polder, G., Vukadinovic, D., et al. (2015) · Machine Vision and Applications · doi:10.1007/s00138-015-0737-3 · Cited by 296 · see citing works
InstanceCut: From Edges to Instances with MultiCut
Kirillov, A., Levinkov, E., Andres, B., Savchynskyy, B., & Rother, C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.774 · Cited by 251 · see citing works
Proposal-Free Network for Instance-Level Object Segmentation
Liang, X., Lin, L., Wei, Y., Shen, X., Yang, J., & Yan, S. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2775623 · Cited by 233 · see citing works
Counting Everyday Objects in Everyday Scenes
Chattopadhyay, P., Vedantam, R., Selvaraju, R. R., Batra, D., & Parikh, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.471 · Cited by 160 · see citing works
Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling
Uhrig, J., Cordts, M., Franke, U., & Brox, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45886-1_2 · Cited by 154 · see citing works
Monocular Object Instance Segmentation and Depth Ordering with CNNs
Zhang, Z., Schwing, A. G., Fidler, S., & Urtasun, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.300 · Cited by 153 · see citing works
Iterative Instance Segmentation
Li, K., Hariharan, B., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.398 · Cited by 152 · see citing works
Layered Object Models for Image Segmentation
Yang, Y., Hallman, S., Ramanan, D., & Fowlkes, C. C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.208 · Cited by 142 · see citing works
The Shape Boltzmann Machine: A Strong Model of Object Shape
Eslami, S. M. A., Heess, N., Williams, C. K. I., & Winn, J. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0669-1 · Cited by 126 · see citing works
Proposal-free Network for Instance-level Object SegmentationPreprint
Liang, X., Wei, Y., Shen, X., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.02636 · Cited by 104 · see citing works
Learning to Count Leaves in Rosette Plants
Giuffrida, M. V., Minervini, M., & Tsaftaris, S. A. (2015) · doi:10.5244/c.29.cvppp.1 · Cited by 97 · see citing works
3-D Histogram-Based Segmentation and Leaf Detection for Rosette Plants
Pape, J.-M., & Klukas, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16220-1_5 · Cited by 93 · see citing works
Instance Segmentation of Indoor Scenes Using a Coverage Loss
Silberman, N., Sontag, D. A., & Fergus, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_40 · Cited by 89 · see citing works
Free-Form Region Description with Second-Order Pooling
Carreira, J., Caseiro, R., Batista, J. P., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2361137 · Cited by 63 · see citing works
Conditional random field with high-order dependencies for sequence labeling and segmentation
Nguyen, C. V., Ye, N., Lee, W. S., et al. (2014) · Journal of Machine Learning Research · doi:10.5555/2627435.2638567 · Cited by 54 · see citing works
Reversible Recursive Instance-Level Object Segmentation
Liang, X., Wei, Y., Shen, X., Jie, Z., Feng, J., Lin, L., & Yan, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.75 · Cited by 53 · see citing works
Banica, D., & Sminchisescu, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298974 · Cited by 45 · see citing works
Multi-leaf tracking from fluorescence plant videos
Yin, X. P., Liu, X., Chen, J., et al. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025081 · Cited by 37 · see citing works
Instance-Level Segmentation with Deep Densely Connected MRFs.Preprint
Zhang, Z., Fidler, S., & Urtasun, R. (2015) · arXiv (Cornell University) · Cited by 25 · see citing works
Shape-aware Instance Segmentation.Preprint
Hayder, Z., He, X., & Salzmann, M. (2016) · arXiv (Cornell University) · Cited by 23 · see citing works
Learning to decompose for object detection and instance segmentationPreprint
Park, E., & Berg, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06449 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Path Aggregation Network for Instance Segmentation
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00913 · Cited by 10,011 · see citing works
Image segmentation using deep learning: A survey
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059968 · Cited by 3,796 · see citing works
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014 · see citing works
High-precision automated reconstruction of neurons with flood-filling networks
Januszewski, M., Kornfeld, J., Li, P. H., Pope, A., Blakely, T., Lindsey, L., Maitin-Shepard, J., Tyka, M., Denk, W., & Jain, V. (2018) · Nature Methods · doi:10.1038/s41592-018-0049-4 · Cited by 380 · see citing works
End-to-End Object Detection with Fully Convolutional Network
Wang, J., Song, L., Li, Z., Sun, H., Sun, J., & Zheng, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01559 · Cited by 235 · see citing works
“Concretely, in the earlier researches, numerous detection frameworks based on recurrent neural networks [43, 35, 26, 32, 36] attempt to produce a set of bounding boxes directly.”
BackgroundMethodObject Detection with Transformers: A Review
Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93 · see citing works
Hierarchical Recurrent Attention Networks for Structured Online Maps
Homayounfar, N., Ma, W.-C., Lakshmikanth, S. K., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00360 · Cited by 71 · see citing works
Translating AI to Clinical Practice: Overcoming Data Shift with Explainability
Choi, Y., Yu, W., Nagarajan, M. B., Teng, P., Goldin, J. G., Raman, S. S., Enzmann, D. R., Kim, G. H. J., & Brown, M. S. (2023) · RadioGraphics · doi:10.1148/rg.220105 · Cited by 65 · see citing works
Recurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62 · see citing works
On the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102 · see citing works
Understanding Self-attention Mechanism via Dynamical System Perspective
Huang, Z., Liang, M., Qin, J., Zhong, S., & Lin, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00136 · Cited by 44 · see citing works
Outdoor RGBD Instance Segmentation With Residual Regretting Learning
Xu, Z., Liu, S., Shi, J., & Lu, C. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2975711 · Cited by 20 · see citing works
- Authors
- Ren, M., & Zemel, R. S.
- Affiliations
- University of Toronto (CA)
- Canadian Institute for Advanced Research (CA)
- ORCID
- Zemel, Richard S.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 293-301
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962676885
- Cited by
- 309
- References
- 47
- Citations by year
- 2026: 3 · 2025: 14 · 2024: 16 · 2023: 28 · 2022: 37 · 2021: 46 · 2020: 66 · 2019: 52 · 2018: 34 · 2017: 12 · 2016: 1
- FWCI
- 13.11
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Samsung
- Intelligence Advanced Research Projects Activity (D16PC00003)
- Interior Business Center
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Keywords
- instance segmentation, recurrent neural networks, attention mechanism, end-to-end learning, object counting, region of interest detection, autonomous driving, visual question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
