CoChat
Sign Up
PreprintCited by 68Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

Domain Robustness in Neural Machine Translation

Open access
Müller, Mathias, Rios, Annette, Sennrich, Rico·2019·arXiv
DOI 10.48550/arxiv.1911.03109arXiv 1911.03109
Open this paper in CoChat — free accountOpen PDF

Abstract

Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustness---the generalization of models to unseen test domains---is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the performance of SMT and NMT models on out-of-domain test sets. We find that in unknown domains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are reported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German to English OPUS data, and German to Romansh (a low-resource setting) we find that several methods improve domain robustness. While those methods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1911.03109

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most relevant literature agrees that translated text is considered a hallucination when the source text is completely disconnected from the translated target.

    “Most relevant literature agrees that translated text is considered a hallucination when the source text is completely disconnected from the translated target [133, 187, 215].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [187]

    Section 3.2.1 (p. 5) describes hallucinated translations as unrelated to the source, supporting the core of the claim. But the paper also requires them to be fluent or at least partially fluent (Table 4, pp. 5–6); disconnection alone is not its definition. The Introduction (p. 2) cites several authors reporting similar behavior, but does not establish agreement across most relevant literature. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Most relevant research papers agree that translated text is considered a hallucination when it is completely disconnected from the source.

    “most of the relevant research papers agree that translated text is considered a hallucination when it is completely disconnected from the source [133, 187].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [187]

    Section 3.2.1 (p. 5) describes hallucinations as translations that are fluent but unrelated to the source content. The Introduction (p. 2) says several authors report this phenomenon, but it does not establish agreement among most relevant papers. The claim also omits the paper’s fluency qualifier. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 36 · Cited by 68

Research works printed but not linked: 30

  1. Arthur, P., Neubig, G., and Nakamura, S. (2016). Incorporating discrete translation lexicons into neural machine translation. In Proceedings of EMNLP 2016 , pages 1557-1567.
  2. Ba, J. and Caruana, R. (2014). Do deep nets really need to be deep? In Advances in neural information processing systems , pages 2654-2662.
  3. Belinkov, Y. and Bisk, Y. (2018). Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations , Vancouver, Canada.
  4. Goodfellow, I., Shlens, J., and Szegedy, C. (2015). Explaining and harnessing adversarial examples. In International Conference on Learning Representations .
  5. Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples (2014). arXiv preprint arXiv:1412.6572 . · arXiv
  6. Hieber, F., Domhan, T., Denkowski, M., Vilar, D., Sokolov, A., Clifton, A., and Post, M. (2018). The Sockeye Neural Machine Translation Toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (AMTA) , pages 200-207.
  7. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 . · arXiv
  8. Jang, E., Gu, S., and Poole, B. (2017). Categorical reparametrization with gumbel-softmax. In Proceedings International Conference on Learning Representations 2017 . OpenReviews.net.
  9. Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Vi´ egas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J. (2017). Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics , 5:339-351.
  10. Kobus, C., Crego, J., and Senellart, J. (2017). Domain control for neural machine translation. In Proceedings of RANLP 2017 , pages 372-378. INCOMA Ltd.
  11. Koehn, P. and Knowles, R. (2017). Six Challenges for Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation , pages 28-39.
  12. Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. In Proceedings of ACL 2018 , pages 66-75.
  13. Li, J. and Jurafsky, D. (2016). Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372 . · arXiv
  14. Li, X., Michel, P., Anastasopoulos, A., Belinkov, Y., Durrani, N., Firat, O., Koehn, P., Neubig, G., Pino, J., and Sajjad, H. (2019). Findings of the First Shared Task on Machine Translation Robustness. In WMT 2019 , pages 91-102, Florence, Italy.
  15. Lison, P. and Tiedemann, J. (2016). OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of LREC 2016 , pages 923-929.
  16. Luong, M.-T. and Manning, C. D. (2015). Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of IWSLT 2015 , Da Nang, Vietnam.
  17. Nguyen, T. and Chiang, D. (2018). Improving Lexical Choice in Neural Machine Translation. In Proceedings of NAACL-HLT 2018 , pages 334-343.
  18. Niu, X., Xu, W., and Carpuat, M. (2019). Bi-directional differentiable input reconstruction for lowresource neural machine translation. In Proceedings of NAACL-HLT 2019 , pages 442-448.
  19. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations .
  20. Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A. (2016). Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP) , pages 582-597. IEEE.
  21. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , pages 311-318, Philadelphia, PA.
  22. Post, M. (2018). A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186-191, Belgium, Brussels.
  23. Scherrer, Y. and Cartoni, B. (2012). The Trilingual ALLEGRA Corpus: Presentation and Possible Use for Lexicon Induction. In Proceedings of LREC-2012 , pages 2890-2896, Istanbul, Turkey.
  24. Sennrich, R., Haddow, B., and Birch, A. (2016a). Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of ACL 2016 , pages 86-96, Berlin, Germany.
  25. Sennrich, R., Haddow, B., and Birch, A. (2016b). Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL 2016 , pages 1715-1725, Berlin, Germany.
  26. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 . · arXiv
  27. Tu, Z., Liu, Y., Shang, L., Liu, X., and Li, H. (2017). Neural Machine Translation with Reconstruction. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence , pages 3097-3103.
  28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is All you Need. In Advances in Neural Information Processing Systems 30 , pages 5998-6008.
  29. Yee, K., Dauphin, Y., and Auli, M. (2019). Simple and effective noisy channel modeling for neural machine translation. In Proceedings of EMNLP-IJCNLP 2019 , pages 5696-5701, Hong Kong, China.
  30. Yu, L., Blunsom, P., Dyer, C., Grefenstette, E., and Kocisk´ y, T. (2017). The neural noisy channel. In 5th International Conference on Learning Representations, ICLR 2017 , Toulon, France.

Authors
Müller, M., Rios, A., & Sennrich, R.
Affiliations
  • University Hospital Regensburg (DE)
  • University of Zurich (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
151-164
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1911.03109
arXiv
1911.03109
OpenAlex
W3105718208
Cited by
68
References
36
Citations by year
2025: 3 · 2024: 3 · 2023: 19 · 2022: 15 · 2021: 22 · 2020: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by-nd)
  • arXiv (Cornell University)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • National Science Foundation
  • European Commission (HORIZON2020, 825460)
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (169888, 105212)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
domain robustness, out-of-domain generalization, neural machine translation, statistical machine translation, translation adequacy, translation fluency, hallucinations, low-resource translation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review