CoChat
Sign Up
Cited by 124Natural Language Processing TechniquesTopic ModelingTranslation Studies and Practices

Hallucinations in Large Multilingual Translation Models

Open access
Guerreiro, Nuno M., Alves, Duarte M., Waldendorf, Jonas, Haddow, Barry, Birch, Alexandra, Colombo, Pierre, Martins, André F. T.·2023·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00615
Open this paper in CoChat — free account

Abstract

Hallucinated translations can severely undermine and raise safety issues when machine translation systems are deployed in the wild. Previous research on the topic focused on small bilingual models trained on high-resource languages, leaving a gap in our understanding of hallucinations in multilingual models across diverse translation scenarios. In this work, we fill this gap by conducting a comprehensive analysis—over 100 language pairs across various resource levels and going beyond English-centric directions—on both the M2M neural machine translation (NMT) models and GPT large language models (LLMs). Among several insights, we highlight that models struggle with hallucinations primarily in low-resource directions and when translating out of English, where, critically, they may reveal toxic patterns that can be traced back to the training data. We also find that LLMs produce qualitatively different hallucinations to those of NMT models. Finally, we show that hallucinations are hard to reverse by merely scaling models trained with the same data. However, employing more diverse models, trained on different data or with different procedures, as fallback systems can improve translation quality and virtually eliminate certain pathologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00615

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reliance on CoT in isolation can inadvertently result in hallucinations, especially in specialist domains such as medicine.

    “We have found that reliance on CoT in isolation can inadvertently result in hallucinations -spurious outputs typically associated with the misapplication of medical terminologies.”

    From MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024) · cites this paper as Guerreiro et al., 2023

    Section 5.4 (p. 10) investigates hallucinations in medical-domain translation, supporting the general point that hallucinations can occur in that setting. It does not examine chain-of-thought (CoT) or attribute hallucinations to misapplied medical terminology. It also reports that hallucination rates were not consistently higher for medical data than for the comparison data. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • LLMs are prone to hallucination, generating plausible yet nonfactual content.

    “Nevertheless, in tandem with these remarkable advancements, concerns have arisen about the tendency of LLMs to generate hallucinations [15, 105], resulting in seemingly plausible yet factually unsupported content.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [105]

    The Introduction (p. 2) reports that ChatGPT, an LLM, produces hallucinations in translation, and Section 2.2 (p. 3) describes some hallucinated translations as fluent but detached from the source text. The paper studies translation, however, and does not establish the broader claim that LLMs generally generate plausible yet factually unsupported content. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
  2. Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondˇ rej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costajussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena AmponsahKaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation , pages 1-88, Online. Association for Computational Linguistics.
  3. Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Franscisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO19: the translation initiative for COvid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020 , Online. Association for Computational Linguistics.
  4. Pierre Andrews, Guillaume Wenzek, Kevin Heffernan, Onur Çelebi, Anna Sun, Ammar Kamran, Yingzhe Guo, Alexandre Mourachko, Holger Schwenk, and Angela Fan. 2022. stopes modular machine translation pipelines. In Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 258-265, Abu Dhabi, UAE. Association for Computational Linguistics.
  5. Meng Cao, Yue Dong, and Jackie Cheung. 2022. Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 33403354, Dublin, Ireland. Association for Computational Linguistics.
  6. David Dale, Elena Voita, Loïc Barrault, and Marta R. Costa-jussà. 2022. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity even better. https://doi.org/10.48550/ARXIV.2212.08597 · DOI
  7. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding backtranslation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
  8. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
  9. Joulin. 2020. Beyond english-centric multilingual machine translation. https://doi.org/10.48550/ARXIV.2010.11125 · DOI
  10. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
  11. Javier Ferrando, Gerard I. Gállego, Belen Alastruey, Carlos Escolano, and Marta R. Costajussà. 2022. Towards opening the black box of neural machine translation: Source and target interpretations of the transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8756-8769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  12. Markus Freitag, Ricardo Rei, Nitika Mathur, Chikiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  13. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores-101 evaluation benchmark for lowresource and multilingual machine translation. Transactions of the Association for Computational Linguistics , 10:522-538.
  14. Nuno M. Guerreiro, Pierre Colombo, Pablo Piantanida, and André F. T. Martins. 2022a. Opti- mal transport for unsupervised hallucination detection in neural machine translation. https://doi.org/10.48550/ARXIV.2212.09631 · DOI
  15. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 . · arXiv
  16. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt. http://arxiv.org/abs/2303.13809 · arXiv
  17. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8348-8359, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  18. Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) , pages 151-164, Virtual. Association for Machine Translation in the Americas.
  19. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) , pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  21. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. ResearchGate preprint .
  22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
  23. Ricardo Rei, José G. C. De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages
  24. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. De Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 86-96, Berlin, Germany. Association for Computational Linguistics.
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.
  27. Guzmán. 2021. Findings of the WMT 2021 shared task on large-scale multilingual machine translation. In Proceedings of the Sixth Conference on Machine Translation , pages 89-99, Online. Association for Computational Linguistics.
  28. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1628-1639, Online. Association for Computational Linguistics.

Cited by

Sorted by citations

Authors
Guerreiro, N. M., Alves, D. M., Waldendorf, J., Haddow, B., Birch, A., Colombo, P., & Martins, A. F. T.
Affiliations
  • University of Lisbon (PT)
  • University of Edinburgh (GB)
  • Université Paris-Saclay (FR)
ORCID
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2023
Volume
11
Pages
1500-1517
ISSN
2307-387X (ISSN-L)
DOI
10.1162/tacl_a_00615
OpenAlex
W4389777735
Cited by
124
References
10
Citations by year
2026: 34 · 2025: 41 · 2024: 29 · 2023: 20
FWCI
11.69
Type
journal article
Language
en
Open access
Licence
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Government of the United Kingdom
  • UK Research and Innovation
  • European Commission (101070631)
  • Grand Équipement National De Calcul Intensif
  • HORIZON EUROPE Framework Programme
  • Fundação para a Ciência e a Tecnologia (UIDB/50008, UIDB/50008/2020, -01-0247-, 50008, LISBOA - 01-0247)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
Keywords
hallucinations, multilingual machine translation, neural machine translation, large language models, low-resource languages, hallucination mitigation, model diversity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review