CoChat
Sign Up
PreprintCited by 5Natural Language Processing TechniquesTopic ModelingSpeech Recognition and Synthesis

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Prevent the Language Model from being Overconfident in Neural Machine Translation

Open access
Miao, Mengqi, Meng, Fandong, Liu, Yijin, Zhou, Xiao-Hua, Zhou, Jie·2021·arXiv
DOI 10.48550/arxiv.2105.11098arXiv 2105.11098
Open this paper in CoChat — free accountOpen PDF

Abstract

The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Language Model (LM) that predicts the next token only based on partial translation. Despite its success, NMT still suffers from the hallucination problem, generating fluent but inadequate translations. The main reason is that NMT pays excessive attention to the partial translation while neglecting the source sentence to some extent, namely overconfidence of the LM. Accordingly, we define the Margin between the NMT and the LM, calculated by subtracting the predicted probability of the LM from that of the NMT model for each token. The Margin is negatively correlated to the overconfidence degree of the LM. Based on the property, we propose a Margin-based Token-level Objective (MTO) and a Margin-based Sentencelevel Objective (MSO) to maximize the Margin for preventing the LM from being overconfident. Experiments on WMT14 English-to-German, WMT19 Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improvements, respectively, compared to the Transformer baseline. The human evaluation further verifies that our approaches improve translation adequacy as well as fluency.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2105.11098

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hallucinated sentences often sound over-confident and frequently contain statistically improbable tokens due to suboptimal decoding strategies.

    “Hallucinated sentences often sound over-confident (Miao et al., 2021; Chen et al., 2022) and frequently contain tokens that are statistically improbable within a given context, primarily due to suboptimal decoding strategies.”

    From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Miao et al., 2021

    The Introduction links hallucinated neural-machine translations to overconfidence of the language-model mechanism: the model favors fluent continuation while neglecting the source sentence. It does not say that the resulting sentences sound over-confident to readers. The citation is attached to that description, not to the later claim about improbable tokens and decoding strategies. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. · arXiv
  2. Christos Baziotis, Barry Haddow, and Alexandra Birch. 2020. Language model prior for low-resource neu- ral machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7622-7634, On- line. Association for Computational Linguistics. · DOI
  3. Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar. Association for Computational Linguistics. · DOI · Paper page
  4. Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019. On the use of BERT for neu- ral machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 108-117, Hong Kong. Association for Com- putational Linguistics. · DOI
  5. Yang Feng, Wanying Xie, Shuhao Gu, Chenze Shao, Wen Zhang, Zhengxin Yang, and Dong Yu. 2019. Modeling fluency and faithfulness for di- verse neural machine translation. arXiv preprint arXiv:1912.00178. · arXiv
  6. Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 5539-5544, Florence, Italy. Asso- ciation for Computational Linguistics. · DOI
  7. C ¸aglar Gülc ¸ehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loïc Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On us- ing monolingual corpora in neural machine transla- tion. CoRR, abs/1503.03535. · arXiv
  8. Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 888-895, Belgium, Brussels. Association for Computational Linguistics. · DOI · Paper page
  9. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel- lart, and Alexander Rush. 2017. OpenNMT: Open- source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
  10. Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic re- lations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI
  11. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceed- ings of the 2004 Conference on Empirical Meth- ods in Natural Language Processing, pages 388- 395, Barcelona, Spain. Association for Computa- tional Linguistics.
  12. Xiang Kong, Zhaopeng Tu, Shuming Shi, Eduard Hovy, and Tong Zhang. 2019. Neural machine trans- lation with adequacy-oriented learning. Proceed- ings of the AAAI Conference on Artificial Intelli- gence, 33(01):6618-6625. · DOI
  13. Thang Luong, Hieu Pham, and Christopher D. Man- ning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pages 1412-1421, Lis- bon, Portugal. Association for Computational Lin- guistics. · DOI · Paper page
  14. Fandong Meng, Zhengdong Lu, Hang Li, and Qun Liu. 2016. Interactive attention for neural machine trans- lation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 2174-2185, Osaka, Japan.
  15. Fandong Meng, Zhaopeng Tu, Yong Cheng, Haiyang Wu, Junjie Zhai, Yuekui Yang, and Di Wang. 2018. Neural machine translation with key-value memory- augmented attention. In Proceedings of IJCAI.
  16. Fandong Meng and Jinchao Zhang. 2019. DTMT: A novel deep transition architecture for neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 224- 231.
  17. Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 955-960, Austin, Texas. Association for Computational Linguistics. · DOI
  18. Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Pro- ceedings of the ACL 2010 Conference Short Papers, pages 220-224, Uppsala, Sweden. Association for Computational Linguistics.
  19. Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine trans- lation. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 151-164, Virtual. Association for Machine Translation in the Ameri- cas.
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Min- nesota. Association for Computational Linguistics. · DOI
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics. · DOI · Paper page
  22. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics. · DOI · Paper page
  23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Asso- ciation for Computational Linguistics. · DOI
  24. Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2017. Cold fusion: Training seq2seq models together with language models. CoRR, abs/1708.06426. · arXiv
  25. Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018. Simple fusion: Return of the language model. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 204-211, Brus- sels, Belgium. Association for Computational Lin- guistics. · DOI
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27:3104-3112.
  27. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.
  28. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85.
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, pages 5998-6008.
  30. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural ma- chine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3544-3552, Online. Association for Computational Linguistics. · DOI · Paper page
  31. Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019a. Improving back-translation with uncertainty-based confidence estimation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 791- 802, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
  32. Wei Wang, Isaac Caswell, and Ciprian Chelba. 2019b. Dynamically composing domain-data selection with clean-data selection by "co-curricular learning" for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 1282-1292.
  33. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural ma- chine translation. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing, pages 1400-1410, Copenhagen, Den- mark. Association for Computational Linguistics. · DOI
  34. Rongxiang Weng, Haoran Wei, Shujian Huang, Heng Yu, Lidong Bing, Weihua Luo, and Jiajun Chen. 2020a. Gret: Global representation enhanced trans- former. AAAI.
  35. Rongxiang Weng, Heng Yu, Xiangpeng Wei, and Wei- hua Luo. 2020b. Towards enhancing faithfulness for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2675-2684, Online. Association for Computational Linguistics. · DOI
  36. Jianhao Yan, Fandong Meng, and Jie Zhou. 2020b. Multi-unit transformers for neural machine transla- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1047-1059, Online.
  37. Baosong Yang, Jian Li, Derek F Wong, Lidia S Chao, Xing Wang, and Zhaopeng Tu. 2019. Context-aware self-attention networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 387- 394.
  38. Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Mod- eling localness for self-attention networks. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 4449- 4458, Brussels, Belgium. Association for Computa- tional Linguistics. · DOI
  39. Boliang Zhang, Ajay Nagesh, and Kevin Knight. 2020. Parallel corpus filtering via pre-trained language models. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 8545-8554, Online. Association for Computa- tional Linguistics. · DOI
  40. Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019a. Bridging the gap between train- ing and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4334- 4343, Florence, Italy.
  41. Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019b. Curriculum learning for domain adaptation in neu- ral machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1903-1915, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  42. Zaixiang Zheng, Shujian Huang, Zhaopeng Tu, Xin- Yu Dai, and Jiajun Chen. 2019. Dynamic past and · DOI

Cited by

Sorted by citations

Metadata

Authors
Miao, M., Meng, F., Liu, Y., Zhou, X.-H., & Zhou, J.
Affiliations
  • Peking University (CN)
  • Tencent (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2105.11098
arXiv
2105.11098
OpenAlex
W3165743422
Cited by
5
References
1
Citations by year
2026: 1 · 2025: 1 · 2023: 1 · 2021: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (12026606)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
neural machine translation, hallucinations, translation adequacy, translation fluency, next-token prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review