Cited by 8,382Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.

Effective Approaches to Attention-based Neural Machine Translation

Open access

Luong, Thang, Pham, Hieu, Manning, Christopher D.·2015·Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

DOI 10.18653/v1/d15-1166

Open this paper in CoChat — free account Open PDF

Abstract

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation.However, there has been little work exploring useful architectures for attention-based NMT.This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout.Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.18653/v1/d15-1166

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Figure 1 Neural machine translation - a stacking recurrent architecture for translating a source sequence A B C D into a target sequence X Y Z . Here, < eos > marks the end of a sentence.CC-BY · From the PDF

Figure 2 Global attentional model - at each time step t , the model infers a variable-length alignment weight vector a t based on the current target state h t and all source states ¯ h s . A global context vector c t is then computed as the weighted average, according to a t , over all the source states.CC-BY · From the PDF

Figure 3 Local attention model - the model first predicts a single aligned position p t for the current target word. A window centered around the source position p t is then used to compute a context vector c t , a weighted average of the source hidden states in the window. The weights a t are inferred from the current target state h t and those source states ¯ h s in the window.CC-BY · From the PDF

Figure 4 Input-feeding approach - Attentional vectors ˜ h t are fed as inputs to the next time steps to inform the model about past alignment decisions.CC-BY · From the PDF

Figure 5 Learning curves - test cost ( ln perplexity) on newstest2014 for English-German NMTs as training progresses.CC-BY · From the PDF

Figure 6 Length Analysis - translation qualities of different systems as sentences become longer.CC-BY · From the PDF

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,393
Statistical phrase-based translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073462 · Cited by 3,265
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 3,174
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1176 · Cited by 1,277
Statistical Phrase-Based Translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · doi:10.21236/ada461156 · Cited by 970

Show all 18 references

Research works printed but not linked: 1

Percy Liang, Ben Taskar, and Dan Klein. 2006. Align- ment by agreement. In NAACL.

Sorted by citations

Items 1,001–1,045 of 1,045

Reliable customer analysis using federated learning and exploring deep-attention edge intelligence
Ahmed, U., Srivastava, G., & Lin, J. C.-W. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.08.028 · Cited by 52
Teaching Machines to Ask Questions
Yao, K., Zhang, L., Luo, T., Tao, L., & Wu, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/632 · Cited by 44
Generating Question-Answer Hierarchies
Krishna, K., & Iyyer, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1224 · Cited by 44
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50
A Holistic Feature Selection Method for Enhanced Short-term Load Forecasting of Power System
Jiang, B., Liu, Y., Geng, H., Wang, Y., Zeng, H., & Ding, J. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3219499 · Cited by 46
PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition
Ngwe, J. L., Lim, K. M., Lee, C. P., Ong, T. S., & Alqahtani, A. (2024) · IEEE Access · doi:10.1109/access.2024.3407108 · Cited by 45
Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ace8be · Cited by 42
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43
Not All Attention Is Needed: Gated Attention Network for Sequence Data
Xue, L., Li, X., & Zhang, N. L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6129 · Cited by 43
Generating Product Descriptions from User Reviews
Novgorodov, S., Guy, I., Elad, G., & Radinsky, K. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313532 · Cited by 41
Reinforced Dynamic Reasoning for Conversational Question Generation
Pan, B., Li, H., Yao, Z., Cai, D., & Sun, H. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1203 · Cited by 40
Interconnected Question Generation with Coreference Alignment and Conversation Flow Modeling
Gao, Y., Li, P., King, I., & Lyu, M. R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1480 · Cited by 44
Developing artificial intelligence models for medical student suturing and knot-tying video-based assessment and coaching
Nagaraj, M. B., Namazi, B., Sankaranarayanan, G., & Scott, D. J. (2023) · Surgical Endoscopy · doi:10.1007/s00464-022-09509-y · Cited by 40
A Semantic Role-based Approach to Open-Domain Automatic Question Generation
Flor, M., & Riordan, B. (2018) · Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/w18-0530 · Cited by 39
SynShine: Improved Fixing of Syntax Errors
Ahmed, T., Ledesma, N. R., & Devanbu, P. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3212635 · Cited by 38
LightSeq: A High Performance Inference Library for Transformers
Wang, X., Xiong, Y., Wei, Y., Wang, M., & Li, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers · doi:10.18653/v1/2021.naacl-industry.15 · Cited by 36
Revisiting Challenges in Data-to-Text Generation with Fact Grounding
Wang, H. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8639 · Cited by 34
Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
Zhou, X., Luo, S., & Wu, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6522 · Cited by 34
Deep learning prediction of chemical-induced dose-dependent and context-specific multiplex phenotype responses and its application to personalized alzheimer’s disease drug repurposing
Wu, Y., Liu, Q., Qiu, Y., & Xie, L. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010367 · Cited by 34
A channel-spatial-temporal attention-based network for vibration-based damage detection
Liao, S., Liu, H., Yang, J., & Ge, Y. (2022) · Information Sciences · doi:10.1016/j.ins.2022.05.042 · Cited by 31
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26
QuickEdit: Editing Text and Translations by Crossing Words Out
Grangier, D., & Auli, M. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1025 · Cited by 20
Shelley: A Crowd-sourced Collaborative Horror Writer
Yanardag, P., Cebrian, M., & Rahwan, I. (2021) · Creativity and Cognition · doi:10.1145/3450741.3465251 · Cited by 20
Thinking Like Transformers
Weiss, G., Goldberg, Y., & Yahav, E. (2021) · arXiv · doi:10.48550/arxiv.2106.06981 · Cited by 18
Extracting Cause of Death from Verbal Autopsy with Deep Learning interpretable methods
Blanco, A., Perez, A., Casillas, A., & Cobos, D. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3005769 · Cited by 17
DEMENTIA: a hybrid attention-based multimodal and multi-task learning framework with expert knowledge for Alzheimer’s disease assessment from speech
Zhang, Z., Wang, T., Hu, Z., Yang, L.-Z., & Li, H. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3509620 · Cited by 17
Question Generation With Doubly Adversarial Nets
Bao, J., Gong, Y., Duan, N., Zhou, M., & Zhao, T. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2859777 · Cited by 14
Fine-grained talking face generation with video reinterpretation
Huang, X., Wang, M., & Gong, M. (2021) · The Visual Computer · doi:10.1007/s00371-020-01982-7 · Cited by 11
Attention Mechanism for Human Motion Prediction
Al-aqel, A. F., & Ali Khan, M. (2020) · 2020 3rd International Conference on Computer Applications & Information Security (ICCAIS) · doi:10.1109/iccais48893.2020.9096777 · Cited by 7
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
Irie, K., Csordás, R., & Schmidhuber, J. (2022) · arXiv · doi:10.48550/arxiv.2202.05798 · Cited by 6
“Obviously, referring to the equations above as attention (or unnormalised attention) emphasizes the relation to the regular softmax normalised dot product attention (Luong et al., 2015; Bahdanau et al., 2015) which is V softmax( K (cid:124) q ) using the notations above.”
Method
Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42
Unsupervised Quality Estimation for Neural Machine Translation
Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., & Specia, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00330 · Cited by 10
Latent Execution for Neural Program Synthesis
Chen, X., Song, D., & Tian, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.00101 · Cited by 2
A hybrid plagiarism detection framework using lexical and semantic similarity with lightweight sentence transformers
Birwadkar, R. (2025) · Journal of Computer and Forensic Sciences · doi:10.5937/jcfs4-64419
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008
“The attention mechanism is initially introduced in RNN-based encoder-decoder architectures (Bahdanau et al., 2015; Luong et al., 2015).”
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
“Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].”
Background
Attention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458
Untitled work
(2018) · doi:10.4230/lipics.cp.2025.27
How Attentive are Graph Attention Networks?
Brody, S., Alon, U., & Yahav, E. (2021) · arXiv · doi:10.48550/arxiv.2105.14491 · Cited by 789
Unsupervised Neural Machine Translation
Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1710.11041 · Cited by 632
Untitled work
(2020) · arXiv (Cornell University) · Cited by 461
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404
Synthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251
SPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78

Authors: Luong, T., Pham, H., & Manning, C. D.
Affiliations: Stanford University (US)
ORCID: Pham, Hieu
Manning, Christopher D.
Venue: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
Publisher: Association for Computational Linguistics
Published: 2015
Pages: 1412-1421
DOI: 10.18653/v1/d15-1166
OpenAlex: W1902237438
Cited by: 8,382
References: 18
Citations by year: 2026: 177 · 2025: 454 · 2024: 586 · 2023: 839 · 2022: 985 · 2021: 1356 · 2020: 1354 · 2019: 1372 · 2018: 783 · 2017: 337 · 2016: 133 · 2015: 5
FWCI: 236.52
Type: conference paper
Language: en
Open access: hybrid
doi.org
Licence: cc-by
Locations: Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Funders: Bloomberg L.P.
Nvidia (Tesla K40)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords: attention mechanism, neural machine translation, local attention, global attention, English-German translation, ensemble learning, dropout

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Effective Approaches to Attention-based Neural Machine Translation

Abstract

Overview

Integrity signals

Figures

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks