The DOI matches its DataCite record; 33 of the 36 references it cites were found in a registry or an index.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Open accessAbstract
Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, LLMs present another challenge: they struggle to accurately represent languages and communities for which there are limited training data 69,94 .”
From Machine culture (2023) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“GPT models and the conventional Neural Machine Translation (NMT) systems are both based on the transformer architecture (Vaswani et al., 2017), but they differ in several aspects.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“For obtaining word to word alignments, we use a multilingual-bert based aligner (Devlin et al., 2019; Dou and Neubig, 2021).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1×
“This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022; Goyal et al., 2022; Wei et al., 2022; Chowdhery et al., 2022).”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 3×
“The emergence of large language models with diverse capabilities, including machine translation, has opened ∗Corresponding author: hanyh@microsoft.com up new possibilities for building more effective translation systems (Brown et al., 2020; Chowdhery et al., 2022).”
BackgroundNo Language Left Behind: Scaling Human-Centered Machine TranslationPreprint
Costa-jussà, M., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.04672 · Cited by 2,176 · cited 1×
“Recent advancements in natural language processing (NLP), particularly the development of largescale language modeling techniques, have brought remarkable improvements in machine translation as well as in other NLP tasks (Fan et al., 2021; Kim et al., 2021; Costa-jussà et al., 2022).”
BackgroundLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 1×
“…learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022; Kojima et al., 2022; Anonymous, 2023a).”
BackgroundMultilingual Denoising Pre-training for Neural Machine Translation
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., & Zettlemoyer, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00343 · Cited by 1,762 · cited 2×
“Hence, in this section, we report results using doc-BLEU (Liu et al., 2020) and doc-COMET metrics as described in §2.4, in addition to our sentence-level metrics.”
MethodInfluentialchrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Proceedings of the Tenth Workshop on Statistical Machine Translation · doi:10.18653/v1/w15-3049 · Cited by 1,220 · cited 2×
“Additionally, we report results using SacreBLEU11 and Chrf (Popović, 2015) for completeness, although we note that these metrics are not extensively used in our analysis.”
MethodLanguage-agnostic BERT Sentence Embedding
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.62 · Cited by 758 · cited 4×
“• Quality: To ensure high quality shots we sort our training data using LaBSE (Feng et al., 2020).”
MethodInfluentialRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 4×
“Analyses of in-context learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022; Kojima et al., 2022; Anonymous, 2023a).”
BackgroundResultInfluentialBeyond English-Centric Multilingual Machine TranslationPreprint
Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Çelebi, O., Wenzek, G., Chaudhary, V., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11125 · Cited by 609
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task
Rei, R., de Souza, J. G. D., Alves, D. M., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.52 · Cited by 554 · cited 2×
“Specifically, we use COMET-22 (wmt22-COMET-da) (Rei et al., 2022a), a reference-based metric that combines direct assessments (DA), sentence-level scores, and word-level tags from Multidimensional Quality Metrics (MQM) error annotations.”
MethodResults of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust
Freitag, M., Rei, R., Mathur, N., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.2 · Cited by 371
Findings of the 2022 Conference on Machine Translation (WMT22)
Kocmi, T., Bawden, R., Bojar, O., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.1 · Cited by 290
In-context Examples Selection for Machine Translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.02437 · Cited by 273 · cited 1×
“In the same vein, Agrawal et al. (2022) use a much smaller model XGLM-7.”
MethodPrompting PaLM for Translation: Assessing Strategies and PerformancePreprint
Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2211.09102 · Cited by 264 · cited 2×
“This is consistent with similar findings from (Vilar et al., 2022) on PALM-540B model.”
BackgroundResultPrompting Large Language Model for Machine Translation: A Case StudyPreprint
Zhang, B., Haddow, B., & Birch, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.07069 · Cited by 184 · cited 2×
“Recently, Zhang et al. (2023) use GLM-130B and show consistent but rather low correlation between the performance of MT and shot selection as compared to random.”
BackgroundNews Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · cited 1×
“This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022; Goyal et al., 2022; Wei et al., 2022; Chowdhery et al., 2022).”
BackgroundWord Alignment by Fine-tuning Embeddings on Parallel Corpora
Dou, Z.-Y., & Neubig, G. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.181 · Cited by 172 · cited 1×
“For obtaining word to word alignments, we use a multilingual-bert based aligner (Devlin et al., 2019; Dou and Neubig, 2021).”
MethodThe unreasonable effectiveness of few-shot learning for machine translationPreprint
García, X., Bansal, Y., Cherry, C., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2302.01398 · Cited by 143 · cited 1×
“…against the noise associated with parallel data, which leads to problems such as the memorization of noisy/atypical (Raunak et al., 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predominant in the parallel data (Garcia et al., 2023).”
BackgroundThe Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142 · cited 3×
“First, not using parallel data might imply that LLMs are protected against the noise associated with parallel data, which leads to problems such as the memorization of noisy/atypical (Raunak et al., 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predomina…”
BackgroundMethodSelf-citationScalable and Efficient MoE Training for Multitask Multilingual ModelsPreprint
Kim, Y. J., Awan, A. A., Muzio, A. F. V., Cruz-Salinas, A. F., Lu, L., Hendy, A., Rajbhandari, S., He, Y., & Awadalla, H. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10465 · Cited by 109
Self-citationThe Effect of Translationese in Machine Translation Test Sets
Zhang, M., & Toral, A. (2019) · Conference on Machine Translation · doi:10.18653/v1/w19-5208 · Cited by 105 · cited 1×
“Second, they have natural source texts and translated target texts, which avoid the problems of using “translationese” testsets with unnatural source texts in the original language that may result in inaccurate evaluation (Zhang and Toral, 2019).”
BackgroundRethinking Document-level Neural Machine Translation
Sun, Z., Wang, M., Zhou, H., Zhao, C., Huang, S., Chen, J., & Li, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.279 · Cited by 62 · cited 2×
“Sun et al. (2020) challenge some of the prior studies and demonstrate that a simple modification of the training to vary the document lengths can significantly enhance the performance of a standard transformer architecture for document-to-document translation.”
ResultLanguage Models are Multilingual Chain-of-Thought Reasoners
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.03057 · Cited by 60 · cited 4×
“We hypothesize that this is due to the large multilingual data proportion in its training data, which is 78% English and 22% for other languages (Shi et al., 2022), while GPT data proportion is only 7% non-English (Brown et al., 2020).”
BackgroundMethodResultInfluentialMeasuring the Cognitive Effort of Literal Translation Processes
Schaeffer, M., & Carl, M. (2014) · HaCaT@EACL · doi:10.3115/v1/w14-0306 · Cited by 43 · cited 1×
“This measurement could also be interpreted as a normalized measure of alignment crossings, which has been shown to correlate with translation non-literalness (Schaeffer and Carl, 2014).”
BackgroundSALTED: A Framework for SAlient Long-Tail Translation Error Detection
Raunak, V., Post, M., & Menezes, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.09988 · Cited by 30 · cited 3×
“Instituting a norm of using multi-dimensional automatic quality measurements (Raunak et al., 2022) can provide very targeted signals on differentiating aspects of translation quality, useful especially when there are competing state-of-the-art approaches.”
BackgroundSelf-citationControlling Machine Translation for Multiple Attributes with Additive Interventions
Schioppa, A., Vilar, D., Sokolov, A., & Filippova, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.535 · Cited by 20 · cited 1×
“We use the nonmonotonicity metric proposed in Schioppa et al. (2021), which computes the deviation from the diagonal in the word to word alignment as the non-monotonicity measure.”
MethodFinding Memo: Extractive Memorization in Constrained Sequence Generation Tasks
Raunak, V., & Menezes, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.12929 · Cited by 17 · cited 2×
“, 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predominant in the parallel data (Garcia et al.”
BackgroundSelf-citationDissecting In-Context Learning of Translations in GPT-3
Raunak, V., Menezes, A., & Awadalla, H. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.61 · Cited by 8
Self-citation
Research works printed but not linked: 5
- Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R Costa-jussà, Cristina España-Bonet, Angela Fan, Christian Federmann, et al. 2021. Findings of the 2021 conference on ma- chine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1- 88.
- Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Feder- mann, Mark Fishel, Alexander Fraser, Markus Fre- itag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, and Christof Monz, ed- itors. 2021. Proceedings of the Sixth Conference on Machine Translation. Association for Computa- tional Linguistics, Online.
- Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, et al. 2022a. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022b. Cometkiwi: Ist-unbabel 2022 sub- mission for the quality estimation shared task. · DOI
Cited by
Sorted by citations
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 1,062
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 916
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233
Metadata
- Authors
- Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., & Awadalla, H. H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.09210
- OpenAlex
- W4321472057
- Cited by
- 202
- References
- 36
- Citations by year
- 2026: 15 · 2025: 48 · 2024: 68 · 2023: 71
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- GPT models, machine translation, ChatGPT, GPT-3.5, prompting strategies, domain shift robustness, document-level translation, low-resource languages, hybrid translation systems, human evaluation of machine translation, translation quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
