CoChat
Sign Up
Cited by 123Natural Language Processing TechniquesTopic ModelingTranslation Studies and Practices

Jointly Learning to Align and Translate with Transformer Models

Open access
Garg, Sarthak, Peitz, Stephan, Nallasamy, Udhyakumar, Paulik, Matthias·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
DOI 10.18653/v1/d19-1453
Open this paper in CoChat — free accountOpen PDF

Abstract

Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias Paulik. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d19-1453

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hallucinations are visible in attention matrices.

    “Tu et al. [257] and Garg et al. [80] show that hallucinations are visible in attention matrices.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [80]

    Section 2.2 describes attention matrices as a way to extract word alignments. The Introduction says alignments might help users diagnose under-translation, but neither passage says hallucinations are visible in attention matrices. Searches for hallucinations and related translation errors found no passage addressing that claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

  • When the model outputs correct translation, the attention mechanism attends to the entire input sequence throughout decoding; however, it tends to concentrate on one point when the model outputs hallucinatory content.

    “When the model outputs correct translation, the attention mechanism attends to the entire input sequence throughout decoding. However, it tends to concentrate on one point when the model outputs hallucinatory content.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [80]

    Section 4.3 says the decoder conditions each next-token prediction on all source tokens, but it does not describe where attention is concentrated during correct translations. The passages found when searching for hallucination and attention concentration do not compare correct translations with hallucinatory output or report attention focusing on one point. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 36 · Cited by 123

Research works printed but not linked: 31

  1. Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018. On the alignment problem in multi-head attention-based neural machine translation. In Con- ference on Empirical Methods in Natural Language Processing, pages 177-185, Brussels, Belgium.
  2. Tamer Alkhouli and Hermann Ney. 2017. Biasing at- tention-based recurrent neural networks using exter- nal alignment information. In Conference on Em- pirical Methods in Natural Language Processing, pages 108-117, Copenhagen, Denmark.
  3. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 1557-1567.
  4. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 confer- ence on machine translation (wmt18). In Confer- ence on Statistical Machine Translation, pages 272- 303, Belgium, Brussels.
  5. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Pa- rameter estimation. Computational Linguistics, 19(2):263-311.
  6. Rajen Chatterjee, Matteo Negri, Marco Turchi, Mar- cello Federico, Lucia Specia, and Frédéric Blain. 2017. Guiding neural machine translation decod- ing with external knowledge. In Proceedings of the Second Conference on Machine Translation, pages 157-168.
  7. Wenhu Chen, Evgeny Matusov, Shahram Khadivi, and Jan-Thorsten Peter. 2016. Guided alignment train- ing for topic-aware neural machine translation. In Association for Machine Translation in the Ameri- cas, pages 121-134, Austin, TX, USA.
  8. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Conference on Empirical Methods in Natu- ral Language Processing, pages 489-500, Brussels, Belgium.
  9. Qin Gao and Stephan Vogel. 2008. Parallel implemen- tations of word alignment tool. In Software Engi- neering, Testing, and Quality Assurance for Natural Language Processing, pages 49-57, Columbus, OH, USA.
  10. Hamidreza Ghader and Christof Monz. 2017. What does attention in neural machine translation pay at- tention to? In International Joint Conference on Natural Language Processing, pages 30-39, Taipei, Taiwan.
  11. Philipp Koehn, Amittai Axelrod, Ra Birch Mayne, Chris Callison-burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description for the 2005 iwslt speech translation evaluation. In Interna- tional Workshop on Spoken Language Translation, pages 68-75, Pittsburgh, PA, USA.
  12. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Annual Meeting of the Association for Computa- tional Linguistics, pages 177-180, Prague, Czech Republic.
  13. Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Pro- ceedings of the First Workshop on Neural Machine Translation, pages 28-39.
  14. Xintong Li, Lemao Liu, Zhaopeng Tu, Shuming Shi, and Max Meng. 2018. Target foresight based atten- tion for neural machine translation. In Conference of the North American Chapter of the Association for Computational Linguistics, pages 1380-1390, New Orleans, LA, USA.
  15. Lemao Liu, Masao Utiyama, Andrew Finch, and Ei- ichiro Sumita. 2016. Neural machine translation with supervised attention. In International Con- ference on Computational Linguistics, pages 3093- 3102, Osaka, Japan.
  16. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine transla- tion. In Conference on Empirical Methods in Natu- ral Language Processing, pages 2283-2288, Austin, TX, USA.
  17. Rada Mihalcea and Ted Pedersen. 2003. An evalua- tion exercise for word alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Transla- tion and Beyond -Volume 3, pages 1-10, Edmonton, Canada.
  18. Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Annual Meeting of the Association for Computational Linguistics, pages 440-447, Hong Kong.
  19. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
  20. Jan-Thorsten Peter, Arne Nix Nix, and Hermann Ney. 2017. Generating alignments using target foresight in attention-based neural machine translation. In Conference of the European Association for Ma- chine Translation, pages 27-36, Prague, Czech Re- public.
  21. Matt Post. 2018. A call for clarity in reporting bleu scores. In Conference on Statistical Machine Trans- lation, pages 186-191, Belgium, Brussels.
  22. Ofir Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. In Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 157-163, Valen- cia, Spain.
  23. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in trans- former-based machine translation. In BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium.
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Annual Meeting of the Associ- ation for Computational Linguistics, pages 1715- 1725, Berlin, Germany.
  25. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguisti- cally-informed self-attention for semantic role label- ing. In Conference on Empirical Methods in Natural Language Processing, pages 5027-5038, Brussels, Belgium.
  26. Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine trans- lation. In Conference on Statistical Machine Trans- lation, pages 26-35, Brussels, Belgium.
  27. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 76-85.
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 1-11, Long Beach, CA, USA.
  29. David Vilar, Maja Popović, and Hermann Ney. 2006. AER: Do we need to improve our alignments? In International Workshop on Spoken Language Trans- lation, pages 205-212, Kyoto, Japan.
  30. Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In International Conference on Com- putational Linguistics, pages 836-841, Copenhagen, Denmark.
  31. Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding Interpretable Attention to Neu- ral Translation Models Improves Word Alignment. arXiv e-prints.

Authors
Garg, S., Peitz, S., Nallasamy, U., & Paulik, M.
Affiliations
Apple (United States) (US)
ORCID
Garg, Sarthak
Venue
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Publisher
Association for Computational Linguistics
Published
2019
Pages
4452-4461
DOI
10.18653/v1/d19-1453
OpenAlex
W2970045405
Cited by
123
References
36
Citations by year
2026: 1 · 2025: 3 · 2024: 7 · 2023: 21 · 2022: 21 · 2021: 48 · 2020: 22
FWCI
8.70
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
Keywords
transformer models, neural machine translation, sentence alignment, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review