CoChat
Sign Up
Cited by 26Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Analysing concatenation approaches to document-level NMT in two different domains

Open access
Scherrer, Yves, Tiedemann, Jörg, Loáiciga, Sharid·2019·Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)
DOI 10.18653/v1/d19-6506
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we investigate how different aspects of discourse context affect the performance of recent neural MT systems.We describe two popular datasets covering news and movie subtitles and we provide a thorough analysis of the distribution of various document-level features in their domains.Furthermore, we train a set of context-aware MT models on both datasets and propose a comparative evaluation scheme that contrasts coherent context with artificially scrambled documents and absent context, arguing that the impact of discourse-aware MT models will become visible in this way.Our results show that the models are indeed affected by the manipulation of the test data, providing a different view on document-level translation quality than absolute sentence-level scores.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d19-6506

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 42

  1. Ruchit Agrawal, Marco Turchi, and Matteo Negri. 2018. Contextual handling in neural machine trans- lation: Look behind, ahead and on both sides. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pages 11-20, Alacant, Spain.
  2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussá, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Trans- lation (WMT19). In Proceedings of the Fourth Con- ference on Machine Translation, pages 128-188, Florence, Italy. Association for Computational Lin- guistics.
  3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phe- nomena in neural machine translation. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1304-1313, New Orleans, Louisiana. Association for Computational Linguis- tics.
  4. Marine Carpuat and Michel Simard. 2012. The trouble with SMT consistency. In Proceedings of the Sev- enth Workshop on Statistical Machine Translation, pages 442-449, Montréal, Canada. Association for Computational Linguistics.
  5. Mauro Cettolo, Girardi Christian, and Federico Mar- cello. 2012. Wit3: Web inventory of transcribed and translated talks. In Proceedings of the Conference of the European Association for Machine Translation, page 261-268.
  6. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for op- timizer instability. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176-181, Portland, Oregon, USA. Association for Computational Linguistics.
  7. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
  8. Federico Fancellu and Bonnie Webber. 2015. Trans- lating negation: Induction, search and model errors. In Proceedings of the Ninth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 21-29, Denver, Colorado, USA. Association for Computational Linguistics. · DOI
  9. Zhengxian Gong, Min Zhang, Chewlim Tan, and Guodong Zhou. 2012. N-gram-based tense mod- els for statistical machine translation. In Proceed- ings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Com- putational Natural Language Learning, EMNLP- CoNLL 2012, pages 276-285, Jeju Island, Korea. Association for Computational Linguistics.
  10. Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of the Student Research Workshop at the 13th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 1-10, Avignon, France. Association for Computational Linguistics.
  11. Liane Guillou and Christian Hardmeier. 2016. Protest: A test suite for evaluating pronouns in machine translation. In Proceedings of the Tenth Interna- tional Conference on Language Resources and Eval- uation (LREC 2016). European Language Resources Association (ELRA).
  12. Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, and Sharid Loáiciga. 2018. A pronoun test suite evaluation of the English-German MT systems at WMT 2018. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 570-577, Belgium, Brussels. Association for Computational Linguistics.
  13. Francisco Guzmán, Shafiq Joty, Lluís Màrquez, and Preslav Nakov. 2014. Using discourse structure im- proves machine translation evaluation. In Proceed- ings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 687-698, Bal- timore, Maryland. Association for Computational Linguistics.
  14. Najeh Hajlaoui and Andrei Popescu-Belis. 2013. As- sessing the accuracy of discourse connective trans- lations: Validation of an automatic metric. In Pro- ceedings of the 14th International Conference on Intelligent Text Processing and Computational Lin- guistics, University of the Aegean, Samos, Greece.
  15. Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical ma- chine translation. In Proceedings of the 7th Interna- tional Workshop on Spoken Language Translation, IWSLT 2010, pages 283-289, Paris, France.
  16. Christian Hardmeier and Liane Guillou. 2016. A graphical pronoun analysis tool for the PROTEST pronoun evaluation test suite. In Proceedings of the 19th Annual Conference of the European Associa- tion for Machine Translation, pages 318-330.
  17. Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2013. Latent anaphora resolution for cross- lingual pronoun prediction. In Proceedings of the 2013 Conference on Empirical Methods in Natu- ral Language Processing, pages 380-391, Seattle, Washington, USA. Association for Computational Linguistics.
  18. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine trans- lation benefit from larger context? In arXiv preprint, arXiv:1704.05135. · arXiv
  19. Marcin Junczys-Dowmunt. 2019. Microsoft Translator at WMT 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation, pages 424-432, Florence, Italy. Association for Computa- tional Linguistics.
  20. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++. In Proceedings of ACL 2018, System Demonstrations, pages 116- 121, Melbourne, Australia. Association for Compu- tational Linguistics.
  21. Ekaterina Lapshinova-Koltunski and Christian Hard- meier. 2017. Discovery of discourse-related lan- guage contrasts through alignment discrepancies in English-German translation. In Proceedings of the Third Workshop on Discourse in Machine Transla- tion, pages 73-81, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
  22. Ekaterina Lapshinova-Koltunski, Sharid Loáiciga, Christian Hardmeier, and Pauline Krielke. 2019. Cross-lingual incongruences in the annotation of coreference. In Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 26-34, Min- neapolis, USA. Association for Computational Lin- guistics.
  23. Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. Euro- pean Language Resources Association (ELRA).
  24. Sharid Loáiciga, Thomas Meyer, and Andrei Popescu- Belis. 2014. English-French verb phrase align- ment in Europarl for tense translation modeling. In Proceedings of the Ninth International Con- ference on Language Resources and Evaluation (LREC'14). European Language Resources Associ- ation (ELRA).
  25. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In Association for Compu- tational Linguistics (ACL) System Demonstrations, pages 55-60.
  26. Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages 3092-3102, Minneapolis, Min- nesota. Association for Computational Linguistics. · DOI
  27. Thomas Meyer and Andrei Popescu-Belis. 2012. Us- ing sense-labeled discourse connectives for statisti- cal machine translation. In Proceedings of the Work- shop on Hybrid Approaches to Machine Translation at EACL 2012, HyTra, pages 129-138, Avignon, France.
  28. Thomas Meyer, Andrei Popescu-Belis, Najeh Hajlaoui, and Andrea Gesmundo. 2012. Machine translation of labeled discourse connectives. In Proceedings of the Tenth Biennial Conference of the Association for Machine Translation in the Americas, AMTA 2012.
  29. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention net- works. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 2947-2954, Brussels, Belgium. Associ- ation for Computational Linguistics.
  30. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the eval- uation of context-aware pronoun translation in neu- ral machine translation. In Proceedings of the Third Conference on Machine Translation: Research Pa- pers, pages 61-72, Belgium, Brussels. Association for Computational Linguistics.
  31. Robert Östling and Jörg Tiedemann. 2016. Effi- cient word alignment with Markov Chain Monte Carlo. Prague Bulletin of Mathematical Linguistics, 106:125-146.
  32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  33. Andrei Popescu-Belis. 2019. Context in neural ma- chine translation: A review of models and evalua- tions.
  34. Anita Ramm and Alexander Fraser. 2016. Model- ing verbal inflection for English to German SMT. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pages 21- 31, Berlin, Germany. Association for Computational Linguistics.
  35. Annette Rios, Mathias Müller, and Rico Sennrich. 2018. The word sense disambiguation test suite at WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 588-596, Belgium, Brussels. Association for Com- putational Linguistics. · DOI
  36. Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambigua- tion in neural machine translation with sense em- beddings. In Proceedings of the Second Conference on Machine Translation, pages 11-19, Copenhagen, Denmark. Association for Computational Linguis- tics. · DOI
  37. Karin Sim Smith. 2017. On integrating discourse in machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 110-121, Copenhagen, Denmark. Association for Computational Linguistics.
  38. Jörg Tiedemann and Yves Scherrer. 2017. Neural ma- chine translation with extended context. In Proceed- ings of the Third Workshop on Discourse in Machine Translation, pages 82-92, Copenhagen, Denmark. Association for Computational Linguistics. · DOI
  39. Don Tuggener. 2016. Incremental coreference resolu- tion for German. Ph.D. thesis, University of Zürich.
  40. Daniel Varga, Péter Halácsy, András Kornai, Viktor Nagy, László Németh, and Viktor Trón. 2005. Par- allel corpora for medium density languages. In Pro- ceedings of RANLP 2005, pages 590-596.
  41. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198-1212, Flo- rence, Italy. Association for Computational Linguis- tics. · DOI
  42. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1264-1274, Melbourne, Australia. Associa- tion for Computational Linguistics.

Cited by

  • LLMs Get Lost In Multi-Turn Conversation

    Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 419

    “The task consists of translating an entire document (10 sentences) from German to English, leveraging paired documents from WMT 2019 on document-level translation [70].”

Metadata

Authors
Scherrer, Y., Tiedemann, J., & Loáiciga, S.
Affiliations
  • University of Helsinki (FI)
  • University of Gothenburg (SE)
Corresponding author
Scherrer, Yves
ORCID
Venue
Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)
Publisher
Association for Computational Linguistics
Published
2019
Pages
51-61
DOI
10.18653/v1/d19-6506
OpenAlex
W2989066524
Cited by
26
References
1
Citations by year
2026: 1 · 2025: 1 · 2024: 3 · 2023: 4 · 2022: 4 · 2021: 7 · 2020: 5 · 2019: 1
FWCI
1.63
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Workshop on Discourse in Machine Translation (DiscoMT) (publishedVersion, cc-by)
  • Työväentutkimus Vuosikirja (submittedVersion)
Funders
  • European Commission (HORIZON2020, 771113, 780069)
  • China Scholarship Council
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
document-level machine translation, discourse context, news translation, film subtitles, translation quality assessment, domain analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review