Cited by 8,382Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.

Effective Approaches to Attention-based Neural Machine Translation

Open access
Luong, Thang, Pham, Hieu, Manning, Christopher D.·2015·Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

Abstract

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation.However, there has been little work exploring useful architectures for attention-based NMT.This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time.We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout.Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d15-1166

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Neural machine translation - a stacking recurrent architecture for translating a source sequence A B C D into a target sequence X Y Z . Here, < eos > marks the end of a sentence.
Figure 1 Neural machine translation - a stacking recurrent architecture for translating a source sequence A B C D into a target sequence X Y Z . Here, < eos > marks the end of a sentence.CC-BY · From the PDF
Figure 2 Global attentional model - at each time step t , the model infers a variable-length alignment weight vector a t based on the current target state h t and all source states ¯ h s . A global context vector c t is then computed as the weighted average, according to a t , over all the source states.
Figure 2 Global attentional model - at each time step t , the model infers a variable-length alignment weight vector a t based on the current target state h t and all source states ¯ h s . A global context vector c t is then computed as the weighted average, according to a t , over all the source states.CC-BY · From the PDF
Figure 3 Local attention model - the model first predicts a single aligned position p t for the current target word. A window centered around the source position p t is then used to compute a context vector c t , a weighted average of the source hidden states in the window. The weights a t are inferred from the current target state h t and those source states ¯ h s in the window.
Figure 3 Local attention model - the model first predicts a single aligned position p t for the current target word. A window centered around the source position p t is then used to compute a context vector c t , a weighted average of the source hidden states in the window. The weights a t are inferred from the current target state h t and those source states ¯ h s in the window.CC-BY · From the PDF
Figure 4 Input-feeding approach - Attentional vectors ˜ h t are fed as inputs to the next time steps to inform the model about past alignment decisions.
Figure 4 Input-feeding approach - Attentional vectors ˜ h t are fed as inputs to the next time steps to inform the model about past alignment decisions.CC-BY · From the PDF
Figure 5 Learning curves - test cost ( ln perplexity) on newstest2014 for English-German NMTs as training progresses.
Figure 5 Learning curves - test cost ( ln perplexity) on newstest2014 for English-German NMTs as training progresses.CC-BY · From the PDF
Figure 6 Length Analysis - translation qualities of different systems as sentences become longer.
Figure 6 Length Analysis - translation qualities of different systems as sentences become longer.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 1

  1. Percy Liang, Ben Taskar, and Dan Klein. 2006. Align- ment by agreement. In NAACL.

Cited by

Sorted by citations

Items 401–500 of 1,045

Metadata

Authors
Luong, T., Pham, H., & Manning, C. D.
Affiliations
Stanford University (US)
Venue
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
Published
2015
Pages
1412-1421
OpenAlex
W1902237438
Cited by
8,382
References
18
Citations by year
2026: 177 · 2025: 454 · 2024: 586 · 2023: 839 · 2022: 985 · 2021: 1356 · 2020: 1354 · 2019: 1372 · 2018: 783 · 2017: 337 · 2016: 133 · 2015: 5
FWCI
236.52
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Funders
  • Bloomberg L.P.
  • Nvidia (Tesla K40)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
attention mechanism, neural machine translation, local attention, global attention, English-German translation, ensemble learning, dropout

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review