Multiple Object Recognition with Visual Attention
Open accessAbstract
We present an attention-based model for recognizing multiple objects in images. The proposed model is a deep recurrent neural network trained with reinforcement learning to attend to the most relevant regions of the input image. We show that the model learns to both localize and recognize multiple objects despite being given only class labels during training. We evaluate the model on the challenging task of transcribing house number sequences from Google Street View images and show that it is both more accurate than the state-of-the-art convolutional networks and uses fewer parameters and less computation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The DRAM method developed in [25] can be used for road accident detection since it can help with multiple object detection from image data.
“The DRAM method developed in [25] can be also used for road accident detection since it can help with multiple object detection from image data.”
From A Review of Machine Learning and IoT in Smart Transportation (2019) · cites this paper as [25]
In [25], DRAM seems less computationally intense than the best CNNs.
“Additionally DRAM seems to be less computationally intense than the best CNNs.”
From A Review of Machine Learning and IoT in Smart Transportation (2019) · cites this paper as [25]
In [25], the DRAM model prevailed over CNNs on multiple digits recognition when tested on Google street view house images.
“the research results indicate that the model prevails over CNNs on multiple digits recognition when tested on Google street view house images.”
From A Review of Machine Learning and IoT in Smart Transportation (2019) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Goodfellow, I. J., Bulatov, Y., Ibarz, J., Arnoud, S., & Shet, V. (2013) · arXiv · doi:10.48550/arxiv.1312.6082 · Cited by 582 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Alexe, Bogdan, Heess, Nicolas, Teh, Yee Whye, and Ferrari, Vittorio. Searching for objects driven by context. In NIPS , 2012. 2
- Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , pp. 1223-1231, 2012a. 9
- Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , pp. 1223-1231, 2012b. 1
- Hochreiter, Sepp and Schmidhuber, J¨ urgen. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 3
- Itti, L., Koch, C., and Niebur, E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence , 20(11):1254-1259, 1998. 2
- Jaderberg, Max, Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227 , 2014a. 1, 2 · arXiv
- Jaderberg, Max, Vedaldi, Andrea, and Zisserman, Andrew. Deep features for text spotting. In Computer Vision-ECCV 2014 , pp. 512-528. Springer, 2014b. 2
- Karpathy, Andrej, Joulin, Armand, and Li, Fei Fei F. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems , pp. 1889-1897, 2014. 1
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoff. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 , pp. 1106-1114, 2012. 1
- Larochelle, Hugo and Hinton, Geoffrey E. Learning to combine foveal glimpses with a third-order boltzmann machine. In NIPS , 2010. 2, 3
- LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, November 1998. 2
- Maes, Francis, Denoyer, Ludovic, and Gallinari, Patrick. Structured prediction with reinforcement learning. Machine learning , 77(2-3):271-301, 2009. 2
- Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, and Kavukcuoglu, Koray. Recurrent models of visual attention. arXiv preprint arXiv:1406.6247 , 2014. 2, 4, 6 · arXiv
- Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, pp. 4, 2011. 5, 7
- Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555 , 2014. 1 · arXiv
- Wang, Tao, Wu, David J, Coates, Adam, and Ng, Andrew Y. End-to-end text recognition with convolutional neural networks. In Pattern Recognition (ICPR), 2012 21st International Conference on , pp. 3304-3308. IEEE, 2012. 2
- Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992. 4
Cited by
Sorted by citations
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_1 · Cited by 25,853 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Self-citationDeep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503 · see citing works
Attention mechanisms in computer vision: A survey
Guo, M.-H., Xu, T.-X., Liu, J.-J., Liu, Z.-N., Jiang, P.-T., Mu, T.-J., Zhang, S.-H., Martin, R. R., Cheng, M.-M., & Hu, S.-M. (2022) · Computational Visual Media · doi:10.1007/s41095-022-0271-y · Cited by 2,566 · see citing works
“[88] proposed a deep recurrent network, similar to RAM [31], capable of processing a multi-resolution crop of the input image, called a glimpse, for multiple object recognition task.”
Artificial intelligence: A powerful paradigm for scientific research
Xu, Y., Liu, X., Cao, X., Huang, C., Liu, E., Qian, S., Liu, X., Wu, Y., Dong, F., Qiu, C.-W., et al. (2021) · The Innovation · doi:10.1016/j.xinn.2021.100179 · Cited by 1,726 · see citing works
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · see citing works
Learning to Track at 100 FPS with Deep Regression Networks
Held, D., Thrun, S., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_45 · Cited by 1,199 · see citing works
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762 · see citing works
GRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637 · see citing works
A Review of Machine Learning and IoT in Smart Transportation
Zantalis, F., Koulouras, G., Karabetsos, S., & Kandris, D. (2019) · Future Internet · doi:10.3390/fi11040094 · Cited by 625 · see citing works
Continuous-variable quantum neural networks
Killoran, N., Bromley, T. R., Arrazola, J. M., Schuld, M., Quesada, N., & Lloyd, S. (2019) · Physical Review Research · doi:10.1103/physrevresearch.1.033063 · Cited by 418 · see citing works
A survey on deep learning-based fine-grained object classification and semantic segmentation
Zhao, B., Feng, J., Wu, X., & Yan, S. (2017) · International Journal of Automation and Computing · doi:10.1007/s11633-017-1053-3 · Cited by 339 · see citing works
Unsupervised deep learning for optical flow estimation
Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., & Zha, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10723 · Cited by 296 · see citing works
KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare
Ma, F., You, Q., Xiao, H., Chitta, R., Zhou, J., & Gao, J. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271701 · Cited by 213 · see citing works
Attention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210 · see citing works
Dynamic Zoom-in Network for Fast Object Detection in Large Images
Gao, M., Yu, R., Li, A., Morariu, V. I., & Davis, L. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00724 · Cited by 148 · see citing works
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136 · see citing works
Multiple Feature Integration for Classification of Thoracic Disease in Chest Radiography
Ho, T. K., & Gwak, J. (2019) · Applied Sciences · doi:10.3390/app9194130 · Cited by 96 · see citing works
Look-Ahead Before You Leap: End-to-End Active Recognition by Forecasting the Effect of Motion
Jayaraman, D., & Grauman, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_30 · Cited by 75 · see citing works
“…is important because, passive recognition systems may have demands such as invariance that are not suited to active recognition and would therefore “distract” the learned features. b) Saliency and attention: Visual saliency and attention are related to active vision [24], [25], [26], [27], [28].”
On the Binding Problem in Artificial Neural Networks
Greff, K., van Steenkiste, S., & Schmidhuber, J. (2020) · arXiv · doi:10.48550/arxiv.2012.05208 · Cited by 102 · see citing works
EPNet: Learning to Exit with Flexible Multi-Branch Network
Dai, X., Kong, X., & Guo, T. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411973 · Cited by 48 · see citing works
LS-Tree: Model Interpretation When the Data Are Linguistic
Chen, J., & Jordan, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5749 · Cited by 14 · see citing works
- Authors
- Ba, J., Mnih, V., & Kavukcuoglu, K.
- Affiliations
- University of Toronto (CA)
- Google DeepMind (United Kingdom) (GB)
- ORCID
- Ba, Jimmy
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2014
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1412.7755
- OpenAlex
- W1484210532
- Cited by
- 808
- References
- 1
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 25 · 2023: 38 · 2022: 61 · 2021: 110 · 2020: 106 · 2019: 144 · 2018: 114 · 2017: 109 · 2016: 58 · 2015: 31 · 2014: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- multiple object recognition, visual attention, attention-based models, deep recurrent neural networks, reinforcement learning, object localization, street view images, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
