Effective Approaches to Attention-based Neural Machine Translation
Open accessAbstract
An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation.However, there has been little work exploring useful architectures for attention-based NMT.This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout.Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Various enhancements to attention have been proposed, such as local attention, supervised attention, hierarchical attention, and self-attention.
“Since then, various enhancements have been proposed, such as local attention (Luong et al. 2015), supervised attention (Mi et al. 2016; Liu et al. 2016), hierarchical attention (Zhao et al. 2018), and self-attention (Vaswani et al. 2017; Yang et al. 2018).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Luong et al. 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
Statistical phrase-based translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073462 · Cited by 3,265 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1176 · Cited by 1,277 · see citing works
Statistical Phrase-Based Translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · doi:10.21236/ada461156 · Cited by 970 · see citing works
On Using Very Large Target Vocabulary for Neural Machine Translation
Jean, S., Cho, K., Memisevic, R., & Bengio, Y. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-1001 · Cited by 839 · see citing works
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · Cited by 785 · see citing works
Addressing the Rare Word Problem in Neural Machine Translation
Luong, M.-T., Sutskever, I., Le, Q. V., Vinyals, O., & Zaremba, W. (2015) · doi:10.3115/v1/p15-1002 · Cited by 626 · see citing works
Liang, P., Taskar, B., & Klein, D. (2006) · doi:10.3115/1220835.1220849 · Cited by 433 · see citing works
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First ResultsPreprint
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 374 · see citing works
Measuring Word Alignment Quality for Statistical Machine Translation
Fraser, A., & Marcu, D. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.3.293 · Cited by 210 · see citing works
N-gram Counts and Language Models from the Common Crawl
Buck, C., Heafield, K., & van Ooyen, B. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3uyduafd26my · Cited by 154 · see citing works
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Percy Liang, Ben Taskar, and Dan Klein. 2006. Align- ment by agreement. In NAACL.
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · see citing works
A review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379 · see citing works
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 3,174 · see citing works
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · see citing works
“The dominant approach to date encodes the input sequence with a series of bi-directional recurrent neural networks (RNN) and generates a variable length output with another set of decoder RNNs, both of which interface via a soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015).”
BackgroundMethodInfluentialSelf-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112 · see citing works
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014 · see citing works
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,771 · see citing works
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F. B., WATTENBERG, M. P., Corrado, G. S., et al. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00065 · Cited by 1,842 · see citing works
Deep Learning--based Text Classification
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad-Khasmakhi, N., Asgari-Chenaghlu, M., & Gao, J. (2021) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,967 · see citing works
Deep learning in bioinformatics
Min, S., Lee, B., & Yoon, S. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw068 · Cited by 1,514 · see citing works
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402 · see citing works
Recurrent Neural Networks for Time Series Forecasting: Current status and future directions
Hewamalage, H., Bergmeir, C., & Bandara, K. (2021) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.06.008 · Cited by 1,371 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
code2vec: learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 1,235 · see citing works
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187 · see citing works
Bidirectional LSTM with attention mechanism and convolutional layer for text classification
Liu, G., & Guo, J. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.078 · Cited by 1,141 · see citing works
Mousavi, S. M., Ellsworth, W. L., Zhu, W., Chuang, L., & Beroza, G. C. (2020) · Nature Communications · doi:10.1038/s41467-020-17591-w · Cited by 1,128 · see citing works
Detecting sequence signals in targeting peptides using deep learning
Armenteros, J. J. A., Salvatore, M., Emanuelsson, O., Winther, O., von HEIJNE, G., Elofsson, A., & Nielsen, H. (2019) · Life Science Alliance · doi:10.26508/lsa.201900429 · Cited by 1,116 · see citing works
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M. A., Thai, T., Moore, K. N., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,129 · see citing works
Aspect Level Sentiment Classification with Deep Memory Network
Tang, D., Qin, B., & Liu, T. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1021 · Cited by 1,078 · see citing works
Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction
Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., & Lee, A. A. (2019) · ACS Central Science · doi:10.1021/acscentsci.9b00576 · Cited by 1,078 · see citing works
Concepts of Artificial Intelligence for Computer-Assisted Drug Discovery
Yang, X. Y., Wang, Y., Byrne, R., Schneider, G., & Yang, S. (2019) · Chemical Reviews · doi:10.1021/acs.chemrev.8b00728 · Cited by 1,045 · see citing works
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1045 · Cited by 1,044 · see citing works
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · see citing works
Temporal pattern attention for multivariate time series forecasting
Shih, S.-Y., Sun, F.-K., & Lee, H.-Y. (2019) · Machine Learning · doi:10.1007/s10994-019-05815-0 · Cited by 965 · see citing works
Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
Wang, K., Peng, X., Yang, J., Meng, D., & Qiao, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2956143 · Cited by 957 · see citing works
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
Yin, W., Schütze, H., Xiang, B., & Zhou, B. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00097 · Cited by 937 · see citing works
Abstractive Sentence Summarization with Attentive Recurrent Neural Networks
Chopra, S., Auli, M., & Rush, A. M. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1012 · Cited by 896 · see citing works
Adversarial Learning for Neural Dialogue Generation
Li, J., Monroe, W., Shi, T., Jean, S., Ritter, A., & Jurafsky, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1230 · Cited by 880 · see citing works
Revisiting Spatial-Temporal Similarity: A Deep Learning Framework for Traffic Prediction
Yao, H., Tang, X., Wei, H., Zheng, G., & Li, Z. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015668 · Cited by 848 · see citing works
Sequence-Level Knowledge Distillation
Kim, Y., & Rush, A. M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1139 · Cited by 836 · see citing works
Neural Sign Language Translation
Camgöz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00812 · Cited by 828 · see citing works
Training Spiking Neural Networks Using Lessons From Deep Learning
Eshraghian, J. K., Ward, M., Neftci, E. O., Wang, X., Lenz, G., Dwivedi, G., Bennamoun, M., Jeong, D. S., & Lu, W. D. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3308088 · Cited by 930 · see citing works
Transfer Learning for Low-Resource Neural Machine Translation
Zoph, B., Yüret, D., May, J., & Knight, K. K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1163 · Cited by 802 · see citing works
An Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 819 · see citing works
Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks
Zhang, C., Li, Q., & Song, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1464 · Cited by 780 · see citing works
Deep Biaffine Attention for Neural Dependency ParsingPreprint
Dozat, T., & Manning, C. D. (2016) · arXiv (Cornell University) · doi:10.4230/oasics.ldk.2021.24 · Cited by 948 · see citing works
A Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Yalta, N., Yamamoto, R., Wang, X., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 766 · see citing works
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_28 · Cited by 753 · see citing works
DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., & Zhang, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11941 · Cited by 779 · see citing works
Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Information · doi:10.3390/info15090517 · Cited by 752 · see citing works
Summarizing Source Code using a Neural Attention Model
Iyer, S. V., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1195 · Cited by 716 · see citing works
Feng, J., Li, Y., Zhang, C., Sun, F., Meng, F., Guo, A., & Jin, D. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186058 · Cited by 718 · see citing works
Modeling Coverage for Neural Machine Translation
Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1008 · Cited by 702 · see citing works
A General Survey on Attention Mechanisms in Deep Learning
Brauwers, G., & Frăsincar, F. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3126456 · Cited by 722 · see citing works
QANet: Combining Local Convolution with Global Self-Attention for Reading ComprehensionPreprint
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1804.09541 · Cited by 1,169 · see citing works
Global Context-Aware Attention LSTM Networks for 3D Action Recognition
Liu, J., Wang, G., Hu, P., Duan, L., & Kot, A. C. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.391 · Cited by 677 · see citing works
Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges
Yang, W., Du, H., Liew, Z. Q., Lim, W. Y. B., Xiong, Z., Niyato, D. T., Chi, X., Shen, X., & Miao, C. (2022) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3223224 · Cited by 699 · see citing works
Serrano, S., & Smith, N. A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1282 · Cited by 690 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · see citing works
“We model the conditional probability us-ing RNN encoder-decoder architecture (Bahdanau et al., 2015; Cho et al., 2014), and adopt the global attention mechanism (Luong et al., 2015a) to make the model focus on certain elements of the input when generating each word during decoding.”
MethodAgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
Yuan, Y., Weng, X., Ou, Y., & Kitani, K. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00967 · Cited by 671 · see citing works
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658 · see citing works
Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence
Liang, H., Tsui, B., Ni, H., Valentim, C. C. S., Baxter, S. L., Liu, G., Cai, W., Kermany, D., Sun, X., Chen, J., et al. (2019) · Nature Medicine · doi:10.1038/s41591-018-0335-9 · Cited by 661 · see citing works
MirrorGAN: Learning Text-To-Image Generation by Redescription
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00160 · Cited by 644 · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundUnified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · see citing works
Attention in Natural Language Processing
Galassi, A., Lippi, M., & Torroni, P. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3019893 · Cited by 678 · see citing works
Video Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z. Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637 · see citing works
Reinforcement Learning for Solving the Vehicle Routing ProblemPreprint
Nazari, M., Oroojlooy, A., Snyder, L., & Takáč, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04240 · Cited by 638 · see citing works
Learning Deep Transformer Models for Machine Translation
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673 · see citing works
Phan, H., Andreotti, F., Cooray, N., Chén, O. Y., & Vos, M. D. (2019) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2019.2896659 · Cited by 631 · see citing works
DeePMD-kit v2: A software package for deep potential models
Zeng, J., Zhang, D., Lu, D., Mo, P., Li, Z., Chen, Y., Rynik, M., Huang, L., Li, Z., Shi, S., et al. (2023) · The Journal of Chemical Physics · doi:10.1063/5.0155600 · Cited by 625 · see citing works
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613 · see citing works
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Chen, Y.-C., & Bansal, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1063 · Cited by 601 · see citing works
Deep neural networks for the evaluation and design of photonic devices
Jiang, J., Chen, M., & Fan, J. A. (2020) · Nature Reviews Materials · doi:10.1038/s41578-020-00260-1 · Cited by 601 · see citing works
FOTS: Fast Oriented Text Spotting with a Unified Network
Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., & Yan, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00595 · Cited by 554 · see citing works
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · see citing works
“Luong et al. [2015] further study the effectiveness of various attention mechanisms with respect to machine translation.”
BackgroundMulti-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
Fırat, O., Cho, K., & Bengio, Y. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1101 · Cited by 570 · see citing works
DeepInf: Social Influence Prediction with Deep Learning
Qiu, J., Tang, J., Ma, H., Dong, Y., Wang, K., & Tang, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220077 · Cited by 587 · see citing works
Deep Learning for Entity Matching
Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., & Raghavendra, V. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3196926 · Cited by 563 · see citing works
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks
Liu, J., Wang, G., Duan, L., Abdiyeva, K., & Kot, A. C. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2785279 · Cited by 605 · see citing works
Deep Learning Methods for Improved Decoding of Linear Codes
Nachmani, E., Marciano, E., Lugosch, L., Gross, W. J., Burshtein, D., & Be'ery, Y. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2788405 · Cited by 539 · see citing works
Attention Branch Network: Learning of Attention Mechanism for Visual Explanation
Fukui, H., Hirakawa, T., Yamashita, T., & Fujiyoshi, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01096 · Cited by 541 · see citing works
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works
Xu, D., Ouyang, W., Wang, X., & Sebe, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00077 · Cited by 541 · see citing works
Learning to decode linear codes using deep learning
Nachmani, E., Be'ery, Y., & Burshtein, D. (2016) · 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton) · doi:10.1109/allerton.2016.7852251 · Cited by 460 · see citing works
Transformation Networks for Target-Oriented Sentiment Classification
Li, X., Bing, L., Lam, W. P., & Shi, B. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1087 · Cited by 538 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Ma, F., Chitta, R., Zhou, J., You, Q., Sun, T., & Gao, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098088 · Cited by 576 · see citing works
Model compression via distillation and quantizationPreprint
Polino, A., Pascanu, R., & Alistarh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05668 · Cited by 648 · see citing works
Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks
Liu, J., Wang, G., Duan, L.-Y., Abdiyeva, K., & Kot, A. C. (2018) · Lancaster EPrints (Lancaster University) · Cited by 531 · see citing works
Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification
Altaheri, H., Muhammad, G., & Alsulaiman, M. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3197419 · Cited by 531 · see citing works
Non-Autoregressive Neural Machine TranslationPreprint
Gu, J., Bradbury, J., Xiong, C., Li, V. O. K., & Socher, R. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.02281 · Cited by 974 · see citing works
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Wang, H., Zhu, Y., Adam, H., Yuille, A., & Chen, L.-C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00542 · Cited by 547 · see citing works
Wireless Image Transmission Using Deep Source Channel Coding With Attention Modules
Xu, J., Ai, B., Chen, W. R., Yang, A., Sun, P., & Rodrigues, M. R. D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3082521 · Cited by 518 · see citing works
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00468 · Cited by 506 · see citing works
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · see citing works
Self-citationMassive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. V. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1151 · Cited by 487 · see citing works
Sequence-to-Sequence Learning as Beam-Search Optimization
Wiseman, S., & Rush, A. M. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1137 · Cited by 485 · see citing works
Seo, S., Huang, J., Yang, H., & Liu, Y. (2017) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3109859.3109890 · Cited by 481 · see citing works
Multi-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01095 · Cited by 483 · see citing works
The multi-modal fusion in visual question answering: a review of attention mechanisms
Lu, S., Liu, M., Yin, L., Yin, Z., Liu, X., & Zheng, W. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1400 · Cited by 478 · see citing works
Modeling Point Clouds With Self-Attention and Gumbel Subset Sampling
Yang, J., Zhang, Q., Ni, B., Li, L., Liu, J., Zhou, M., & Tian, Q. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00344 · Cited by 465 · see citing works
Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey
He, K., Kim, D. S., & Asghar, M. R. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2022.3233793 · Cited by 494 · see citing works
Dynamic Coattention Networks For Question AnsweringPreprint
Xiong, C., Zhong, V., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01604 · Cited by 827 · see citing works
- Authors
- Luong, T., Pham, H., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
- Published
- 2015
- Pages
- 1412-1421
- OpenAlex
- W1902237438
- Cited by
- 8,382
- References
- 18
- Citations by year
- 2026: 177 · 2025: 454 · 2024: 586 · 2023: 839 · 2022: 985 · 2021: 1356 · 2020: 1354 · 2019: 1372 · 2018: 783 · 2017: 337 · 2016: 133 · 2015: 5 · 2012: 1
- FWCI
- 236.52
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- Bloomberg L.P.
- Nvidia (Tesla K40)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- attention mechanism, neural machine translation, local attention, global attention, English-German translation, ensemble learning, dropout
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
