How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Open accessAbstract
Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs struggle to accurately represent languages and communities for which there are limited training data.
“However, LLMs present another challenge: they struggle to accurately represent languages and communities for which there are limited training data 69,94 .”
From Machine culture (2023) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“GPT models and the conventional Neural Machine Translation (NMT) systems are both based on the transformer architecture (Vaswani et al., 2017), but they differ in several aspects.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“For obtaining word to word alignments, we use a multilingual-bert based aligner (Devlin et al., 2019; Dou and Neubig, 2021).”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1× · see citing works
“This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022; Goyal et al., 2022; Wei et al., 2022; Chowdhery et al., 2022).”
BackgroundChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 3× · see citing works
“The emergence of large language models with diverse capabilities, including machine translation, has opened ∗Corresponding author: hanyh@microsoft.com up new possibilities for building more effective translation systems (Brown et al., 2020; Chowdhery et al., 2022).”
BackgroundNo Language Left Behind: Scaling Human-Centered Machine TranslationPreprint
Costa-jussà, M., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.04672 · Cited by 2,176 · cited 1× · see citing works
“Recent advancements in natural language processing (NLP), particularly the development of largescale language modeling techniques, have brought remarkable improvements in machine translation as well as in other NLP tasks (Fan et al., 2021; Kim et al., 2021; Costa-jussà et al., 2022).”
BackgroundLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 1× · see citing works
“…learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022; Kojima et al., 2022; Anonymous, 2023a).”
BackgroundMultilingual Denoising Pre-training for Neural Machine Translation
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., & Zettlemoyer, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00343 · Cited by 1,608 · cited 2× · see citing works
“Hence, in this section, we report results using doc-BLEU (Liu et al., 2020) and doc-COMET metrics as described in §2.4, in addition to our sentence-level metrics.”
MethodInfluentialBeyond English-Centric Multilingual Machine TranslationPreprint
Fan, A., Bhosale, S., Schwenk, H., Ma, Z.-Y., El-Kishky, A., Goyal, S., Baines, M., Çelebi, O., Wenzek, G., Chaudhary, V., et al. (2020) · Journal of machine learning research · doi:10.48550/arxiv.2010.11125 · Cited by 1,208 · see citing works
chrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3049 · Cited by 1,026 · cited 2× · see citing works
“Additionally, we report results using SacreBLEU11 and Chrf (Popović, 2015) for completeness, although we note that these metrics are not extensively used in our analysis.”
MethodLanguage-agnostic BERT Sentence Embedding
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.62 · Cited by 758 · cited 4× · see citing works
“• Quality: To ensure high quality shots we sort our training data using LaBSE (Feng et al., 2020).”
MethodInfluentialRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 4× · see citing works
“Analyses of in-context learning have revealed how the implicit zero-shot performance of LLMs might be higher than their observed zero-shot performance, with the demonstrations within in-context learning themselves providing only limited learning signals (Min et al., 2022; Kojima et al., 2022; Anonymous, 2023a).”
BackgroundResultInfluentialCOMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task
Rei, R., de Souza, J. G. D., Alves, D. M., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.52 · Cited by 554 · cited 2×
“Specifically, we use COMET-22 (wmt22-COMET-da) (Rei et al., 2022a), a reference-based metric that combines direct assessments (DA), sentence-level scores, and word-level tags from Multidimensional Quality Metrics (MQM) error annotations.”
MethodPrompting Large Language Model for Machine Translation: A Case StudyPreprint
Zhang, B., Haddow, B., & Birch, A. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.07069 · Cited by 452 · cited 2× · see citing works
“Recently, Zhang et al. (2023) use GLM-130B and show consistent but rather low correlation between the performance of MT and shot selection as compared to random.”
BackgroundResults of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust
Freitag, M., Rei, R., Mathur, N., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.2 · Cited by 371
Findings of the 2022 Conference on Machine Translation (WMT22)
Kocmi, T., Bawden, R., Bojar, O., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.1 · Cited by 290
In-context Examples Selection for Machine Translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.02437 · Cited by 273 · cited 1×
“In the same vein, Agrawal et al. (2022) use a much smaller model XGLM-7.”
MethodPrompting PaLM for Translation: Assessing Strategies and PerformancePreprint
Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2211.09102 · Cited by 264 · cited 2× · see citing works
“This is consistent with similar findings from (Vilar et al., 2022) on PALM-540B model.”
BackgroundResultNews Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · cited 1× · see citing works
“This few-shot paradigm has demonstrated strong performance across multiple natural language processing (NLP) tasks (Ouyang et al., 2022; Goyal et al., 2022; Wei et al., 2022; Chowdhery et al., 2022).”
BackgroundWord Alignment by Fine-tuning Embeddings on Parallel Corpora
Dou, Z.-Y., & Neubig, G. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.181 · Cited by 147 · cited 1× · see citing works
“For obtaining word to word alignments, we use a multilingual-bert based aligner (Devlin et al., 2019; Dou and Neubig, 2021).”
MethodThe unreasonable effectiveness of few-shot learning for machine translationPreprint
García, X., Bansal, Y., Cherry, C., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2302.01398 · Cited by 143 · cited 1× · see citing works
“…against the noise associated with parallel data, which leads to problems such as the memorization of noisy/atypical (Raunak et al., 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predominant in the parallel data (Garcia et al., 2023).”
BackgroundThe Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142 · cited 3× · see citing works
“First, not using parallel data might imply that LLMs are protected against the noise associated with parallel data, which leads to problems such as the memorization of noisy/atypical (Raunak et al., 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predomina…”
BackgroundMethodSelf-citationThe Effect of Translationese in Machine Translation Test Sets
Zhang, M., & Toral, A. (2019) · Conference on Machine Translation · doi:10.18653/v1/w19-5208 · Cited by 105 · cited 1× · see citing works
“Second, they have natural source texts and translated target texts, which avoid the problems of using “translationese” testsets with unnatural source texts in the original language that may result in inaccurate evaluation (Zhang and Toral, 2019).”
BackgroundLanguage Models are Multilingual Chain-of-Thought Reasoners
Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.03057 · Cited by 60 · cited 4× · see citing works
“We hypothesize that this is due to the large multilingual data proportion in its training data, which is 78% English and 22% for other languages (Shi et al., 2022), while GPT data proportion is only 7% non-English (Brown et al., 2020).”
BackgroundMethodResultInfluentialScalable and Efficient MoE Training for Multitask Multilingual ModelsPreprint
Kim, Y. J., Awan, A. A., Muzio, A. F. V., Cruz-Salinas, A. F., Lu, L., Hendy, A., Rajbhandari, S., He, Y., & Awadalla, H. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10465 · Cited by 57 · see citing works
Self-citationRethinking Document-level Neural Machine Translation
Sun, Z., Wang, M., Zhou, H., Zhao, C., Huang, S., Chen, J., & Li, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.279 · Cited by 45 · cited 2× · see citing works
“Sun et al. (2020) challenge some of the prior studies and demonstrate that a simple modification of the training to vary the document lengths can significantly enhance the performance of a standard transformer architecture for document-to-document translation.”
ResultMeasuring the Cognitive Effort of Literal Translation Processes
Schaeffer, M., & Carl, M. (2014) · HaCaT@EACL · doi:10.3115/v1/w14-0306 · Cited by 43 · cited 1×
“This measurement could also be interpreted as a normalized measure of alignment crossings, which has been shown to correlate with translation non-literalness (Schaeffer and Carl, 2014).”
BackgroundSALTED: A Framework for SAlient Long-Tail Translation Error Detection
Raunak, V., Post, M., & Menezes, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.09988 · Cited by 30 · cited 3×
“Instituting a norm of using multi-dimensional automatic quality measurements (Raunak et al., 2022) can provide very targeted signals on differentiating aspects of translation quality, useful especially when there are competing state-of-the-art approaches.”
BackgroundSelf-citationControlling Machine Translation for Multiple Attributes with Additive Interventions
Schioppa, A., Vilar, D., Sokolov, A., & Filippova, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.535 · Cited by 17 · cited 1× · see citing works
“We use the nonmonotonicity metric proposed in Schioppa et al. (2021), which computes the deviation from the diagonal in the word to word alignment as the non-monotonicity measure.”
MethodFinding Memo: Extractive Memorization in Constrained Sequence Generation Tasks
Raunak, V., & Menezes, A. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.12929 · Cited by 17 · cited 2×
“, 2021) or low-quality samples (Raunak and Menezes, 2022) and biases towards particular language characteristics predominant in the parallel data (Garcia et al.”
BackgroundSelf-citationDissecting In-Context Learning of Translations in GPT-3
Raunak, V., Menezes, A., & Awadalla, H. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.61 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
3 of the 36 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R Costa-jussà, Cristina España-Bonet, Angela Fan, Christian Federmann, et al. 2021. Findings of the 2021 conference on ma- chine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1- 88.
- Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Feder- mann, Mark Fishel, Alexander Fraser, Markus Fre- itag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, and Christof Monz, ed- itors. 2021. Proceedings of the Sixth Conference on Machine Translation. Association for Computa- tional Linguistics, Online.
- Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, et al. 2022a. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022b. Cometkiwi: Ist-unbabel 2022 sub- mission for the quality estimation shared task. · DOI
Cited by
Sorted by citations
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 759 · see citing works
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 274 · see citing works
Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · see citing works
Lai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233 · see citing works
Amin, M. M., Cambria, E., & Schuller, B. W. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3254179 · Cited by 162 · see citing works
“Hendy et al. 8 explore the capabilities of GPT language models (including ChatGPT) in machine translation.”
BackgroundDocument-Level Machine Translation with Large Language Models
Wang, L., Lyu, C., Ji, T., Zhang, Z., Yu, D., Shi, S., & Tu, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1036 · Cited by 155 · see citing works
“Therefore, we advocate for organizations like OpenAI to provide greater transparency to aid researchers in such explorations.”
MethodRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149 · see citing works
“1 Large Language Model Large Language Models (LLMs) have achieved impressive performance on a wide range of natural language processing (NLP) tasks, including machine translation [8, 17, 34], summarization [12, 15, 34], sentiment analysis [48, 49], and question-answering [31, 34].”
BackgroundBrinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
“In sum, recommender systems influence cultural evolution by rewiring our social networks and modifying information flows such that they can substantially influence the dynamics of cultural markets 47–49 .”
BackgroundLarge Language Models for UAVs: Current State and Pathways to the Future
Javaid, S., Fahim, H., He, B., & Saeed, N. (2024) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2024.3446799 · Cited by 148 · see citing works
“GPT-2 and GPT-3 used an unsupervised learning model, demonstrating exceptional capabilities in learning from large datasets [87].”
BackgroundLarge Language Models Are State-of-the-Art Evaluators of Translation Quality
Kocmi, T., & Federmann, C. (2023) · arXiv · doi:10.48550/arxiv.2302.14520 · Cited by 122 · see citing works
“This confirms the findings of Hendy et al. (2023) who demonstrated superior translation capabilities with Davinci-003 over all other previous GPT variants.”
BackgroundResultInfluentialHallucinations in Large Multilingual Translation Models
Guerreiro, N. M., Alves, D. M., Waldendorf, J., Haddow, B., Birch, A., Colombo, P., & Martins, A. F. T. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00615 · Cited by 124 · see citing works
Artificial intelligence and posthumanist translation: ChatGPT versus the translator
Lee, T. K. (2023) · Applied Linguistics Review · doi:10.1515/applirev-2023-0122 · Cited by 104 · see citing works
Exploring Human-Like Translation Strategy with Large Language Models
He, Z., Liang, T., Jiao, W., Zhang, Z., Yang, Y., Wang, R., Tu, Z., Shi, S., & Wang, X. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00642 · Cited by 124 · see citing works
“These neural-based learned metrics show superiority over string-based metrics like BLEU [13, 31, 4] and have been adopted broadly by LLM-based translation literature [44, 22, 15, 50].”
BackgroundMethodA Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 117 · see citing works
Prompting PaLM for Translation: Assessing Strategies and Performance
Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.859 · Cited by 125 · see citing works
Towards Making the Most of ChatGPT for Machine Translation
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.373 · Cited by 84 · see citing works
Lu, Q., Qiu, B., Ding, L., Xie, L., & Tao, D. (2023) · Preprints.org · doi:10.20944/preprints202303.0255.v1 · Cited by 71 · see citing works
Huang, H., Tang, T., Zhang, D., Zhao, X., Song, T., Xia, Y., & Wei, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.826 · Cited by 75 · see citing works
A Survey of Text Watermarking in the Era of Large Language Models
Liu, A., Pan, L., Lu, Y., Li, J., Hu, X., Zhang, X., Wen, L., King, I., Xiong, H., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3691626 · Cited by 87 · see citing works
“The taxonomy of text watermarking algorithms is also introduced in this section.”
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61 · see citing works
Karpinska, M., & Iyyer, M. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.41 · Cited by 69 · see citing works
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Khondaker, M. T. I., Waheed, A., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.16 · Cited by 79 · see citing works
Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and BardPreprint
Borji, A., & Mohammadian, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476855 · Cited by 53 · see citing works
ChatGPT MT: Competitive for High- (but Not Low-) Resource Languages
Robinson, N., Ogayo, P., Mortensen, D. R., & Neubig, G. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.40 · Cited by 58 · see citing works
Zhang, X., Rajabi, N., Duh, K., & Koehn, P. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.43 · Cited by 47 · see citing works
“Both Hendy et al. (2023) and Moslem et al. (2023) underscore the superiority of GPT3 (Brown et al., 2020), GPT3.”
BackgroundRecent Advances in Interactive Machine Translation With Large Language Models
Wang, Y., Zhang, J., Shi, T., Deng, D., Tian, Y., & Matsumoto, T. (2024) · IEEE Access · doi:10.1109/access.2024.3487352 · Cited by 50 · see citing works
Li, J., Zhou, H., Huang, S., Cheng, S., & Chen, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00655 · Cited by 61 · see citing works
“The emergence of Large Pretrained Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023) has revolutionized the research of machine translation (Hendy et al., 2023; Garcia et al., 2023).”
BackgroundLeveraging GPT-4 for Automatic Translation Post-Editing
Raunak, V., Sharaf, A., Wang, Y., Awadalla, H. H., & Menezes, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.804 · Cited by 45 · see citing works
Self-citationKocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., et al. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.1 · Cited by 39 · see citing works
“Due to the closed-source nature of commercial tools, it is hard to know the exact reasons for these findings, although they confirm previous observations that GPT models have difficulties with under-represented languages (Hendy et al., 2023).”
MethodResultGlobal evidence of human well-being and biodiversity impacts of natural climate solutions
Chang, C. H., Erbaugh, J. T., Fajardo, P., Lu, L., Molnár, I. D., Papp, D., Robinson, B. E., Austin, K. G., Castro, M., Cheng, S. H., et al. (2024) · Nature Sustainability · doi:10.1038/s41893-024-01454-z · Cited by 37 · see citing works
Remadi, A., Hage, K. E., Hobeika, Y., & Bugiotti, F. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102313 · Cited by 36 · see citing works
Large language models "ad referendum": How good are they at machine translation in the legal domain?
Brivá-Iglesias, V., Doğru, G., & Camargo, J. L. C. (2024) · MonTi Monografías de Traducción e Interpretación · doi:10.6035/monti.2024.16.02 · Cited by 38 · see citing works
Kwok, H. L., Shi, Y., Xu, H., Li, D., & Liu, K. (2025) · System · doi:10.1016/j.system.2025.103618 · Cited by 34 · see citing works
Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
Qin, L., Chen, Q., Wei, F., Huang, S., & Che, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.163 · Cited by 50 · see citing works
Language inclusion in ecological systematic reviews and maps: Barriers and perspectives
Hannah, K., Haddaway, N., Fuller, R. A., & Amano, T. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1699 · Cited by 42 · see citing works
“34,36 However, more testing and development is needed before it will be able to be widely utilised.”
GPT for Translation: A Systematic Literature Review
Chan, V. W. M., & Tang, W. K.-W. (2024) · SN Computer Science · doi:10.1007/s42979-024-03340-z · Cited by 33 · see citing works
“The results revealed that some MT evaluation metrics were frequently employed and combined for assessing the translation quality, such as BLEU’s score [2, 32, 34], ChrF [6, 35], and COMET [1, 36].”
Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., & Lan, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.263 · Cited by 37 · see citing works
An Evaluation of ChatGPT's Translation Accuracy Using BLEU Score
Ghassemiazghandi, M. (2024) · Theory and Practice in Language Studies · doi:10.17507/tpls.1404.07 · Cited by 30 · see citing works
“Generative Pre-trained Transformer (GPT), a renowned advanced language model created by OpenAI, has acquired considerable scrutiny for its capacity to comprehend and produce coherent and logical text (Hendy et al., 2023; Sahari et al., 2024).”
BackgroundMcKinney, S. (2024) · Journal of Deliberative Democracy · doi:10.16997/jdd.1556 · Cited by 27 · see citing works
TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties
Kadaoui, K., Magdy, S. M., Waheed, A., Khondaker, M. T. I., El-Shangiti, A. O., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.6 · Cited by 39 · see citing works
PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection
Yang, T., Shi, T., Wan, F., Quan, X., Wang, Q., Wu, B., & Wu, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.216 · Cited by 41 · see citing works
A survey of dialogic emotion analysis: Developments, approaches and perspectives
Gan, C., Zheng, J., Zhu, Q., Cao, Y., & Zhu, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110794 · Cited by 27 · see citing works
Language barriers in conservation: consequences and solutions
Amano, T., & Berdejo‐Espinola, V. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2024.11.003 · Cited by 28 · see citing works
Domain Terminology Integration into Machine Translation: Leveraging Large Language Models
Moslem, Y., Romani, G., Molaei, M., Kelleher, J. D., Haque, R., & Way, A. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.82 · Cited by 26 · see citing works
“…2023), Llama 2 (Touvron et al., 2023), and Jais (Sengupta et al., 2023) to mention just a few, researchers have been exploring the capabilities of these models for a number of tasks including MT (Bawden and Yvon, 2023; Hendy et al., 2023; Jiao et al., 2023; Moslem et al., 2023; Vilar et al., 2023).”
BackgroundLow-Resource Neural Machine Translation: A Systematic Literature Review
Yazar, B. K., Şahi̇n, D. Ö., & Kılıç, E. (2023) · IEEE Access · doi:10.1109/access.2023.3336019 · Cited by 24 · see citing works
“There are studies in the literature where LLMs with known success such as ChatGPT, GPT-4, etc. are used for MT tasks [66], [67].”
BackgroundFinSentGPT: A universal financial sentiment engine?
Ardekani, A. M., Bertz, J., Bryce, C., Dowling, M., & Long, S. (2024) · International Review of Financial Analysis · doi:10.1016/j.irfa.2024.103291 · Cited by 24 · see citing works
Fu, L., & Liu, L. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03726-7 · Cited by 24 · see citing works
“Although its architecture differs from the traditional encoder-decoder blueprint of NMT, GPT models, including ChatGPT 3.5, demonstrate promising translation capabilities (Hendy et al., 2023).”
InfluentialProposal for a Triple Bottom Line for Translation Automation and Sustainability
Moorkens, J., Castilho, S., Gaspari, F., Toral, A., & Popović, M. (2024) · The Journal of Specialised Translation · doi:10.26034/cm.jostrans.2024.4706 · Cited by 24 · see citing works
Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative StudyPreprint
Keraghel, I., Morbieu, S., & Nadif, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10825 · Cited by 78 · see citing works
ChatGPT translation vs. human translation: an examination of a literary text
Rousan, R. M. A., Jaradat, R., & Malkawi, M. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2472916 · Cited by 23 · see citing works
“…concerned with evaluating the St. therefore, SP is zero in the current study. this tool was used by several researchers such as Wang et al. (2023), hendy et al. (2023) and Katarina et al. (2021) to evaluate the performance of Mt. according to them, this tool is reliable since it covers a huge…”
InfluentialZayyanu, Z. M. (2024) · Computer Science & Engineering An International Journal · doi:10.5121/cseij.2024.14302 · Cited by 20 · see citing works
“GPT-3 has been evaluated for translating specialised Arabic text to English and has shown generally comprehensible translations but struggles with capturing nuances and cultural context, (Hendy., A, et al., 2023).”
BackgroundTowards Making the Most of ChatGPT for Machine Translation
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2023) · arXiv · doi:10.48550/arxiv.2303.13780 · Cited by 20 · see citing works
Do GPTs Produce Less Literal Translations?
Raunak, V., Menezes, A., Post, M., & Hassan, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.90 · Cited by 24 · see citing works
Self-citationTranslators’ trust and distrust in the times of GenAI
Ginel, M. I. R., & Moorkens, J. (2025) · Translation Studies · doi:10.1080/14781700.2025.2507594 · Cited by 18 · see citing works
Krüger, R. (2023) · Lebende Sprachen · doi:10.1515/les-2023-0024 · Cited by 24 · see citing works
“Similarly to Hendy et al. (2023), Moslem et al. (2023:229) find diminishing returns in increasing the number of translation examples in a prompt.”
InfluentialParty politics in transport policy with a large language model
Yun, H., & Lee, E. H. (2025) · Transport Policy · doi:10.1016/j.tranpol.2025.06.021 · Cited by 16 · see citing works
Accuracy of Artificial Intelligence vs Professionally Translated Discharge Instructions
Martos, M., Fields, B., Finlayson, S. G., Hartell, N., Kim, T. Y., Larimer, E. L., Lau, J. J., Lin, Y.-H., Salaguinto, T., Tran, N., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.32312 · Cited by 17 · see citing works
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Alves, D., Guerreiro, N., Alves, J., Pombal, J., Rei, R., de Souza, J. G. C., Colombo, P., & Martins, A. F. T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.744 · Cited by 26 · see citing works
Neural Machine Translation for Low-Resource Languages from a Chinese-centric Perspective: A Survey
Zhang, J., Su, K., Li, H., Mao, J., Tian, Y., Wen, F., Guo, C., & Matsumoto, T. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3665244 · Cited by 21 · see citing works
A Survey on Failure Analysis and Fault Injection in AI Systems
Yu, G., Tan, G., Huang, H., Zhang, Z., Chen, P., Natella, R., Zheng, Z., & Lyu, M. R.-T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3732777 · Cited by 18 · see citing works
“Large Language Models (LLMs), in particular, have proven to be a game changer, propelling AI to unprecedented heights and facilitating a myriad of applications in fields such as software engineering [141, 192, 196] and human language translation [8, 59, 85].”
Lei, B., Ding, C., Chen, L., et al. (2023) · IEEE High Performance Extreme Computing Conference · doi:10.1109/hpec58863.2023.10363534 · Cited by 15 · see citing works
“With the rise of large language models (LLMs), particularly transformer-based models such as GPT, the field of natural language processing has seen unprecedented advancements, offering successful solutions to complex tasks such as machine translation [8].”
BackgroundThe Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education
Krause, S., Panchal, B. H., & Ubhe, N. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_24 · Cited by 35 · see citing works
The utility of the Living Planet Index as a policy tool and for measuring nature recovery
McRae, L., Cornford, R., Marconi, V., Puleston, H., Ledger, S. E. H., Deinet, S., Oppenheimer, P., Hoffmann, M., & Freeman, R. (2025) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0207 · Cited by 16 · see citing works
“…reducing the bottlenecks associated with manual data searches [105]; open-source large language tools such as ChatGPT could also be harnessed to do this, although their models are optimized for English and are likely to be more effective for languages with large available data resources [106].”
Banat, M., & Adla, Y. A. (2023) · Journal of Translation and Language Studies · doi:10.48185/jtls.v4i2.762 · Cited by 14 · see citing works
“In their research titled "A Comprehensive Evaluation of Generative Pre-trained Transformer (GPT) Models for Machine Translation" [1], the authors conducted a detailed examination of the efficacy of GPT models in machine translation.”
BackgroundMethodMahdi, H. S., & Sahari, Y. (2024) · Journal of Pedagogical Research · doi:10.33902/jpr.202423345 · Cited by 14 · see citing works
Li, X. (2024) · International Journal of Education and Humanities · doi:10.58557/(ijeh).v4i2.213 · Cited by 14 · see citing works
“Developed by OpenAI, it was trained to respond to users’ requests wisely in great detail, including accomplishing translation tasks, and is much more capable than any other previous AI of generating coherent and context-aware text (Hendy et al., 2023).”
BackgroundMethodGPT and Translation: A Systematic Review
Chan, V. W. M., & Tang, W. K.-W. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00021 · Cited by 17 · see citing works
“The substantial advancements in artificial intelligence (AI), particularly the development of natural language processing (NLP) and large language models (LLM), have reshaped translation practice and led to remarkable improvements in machine translation (MT) [1].”
BackgroundTian, S., Wang, D., Wang, J., & Zhong, W. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2534269 · Cited by 16 · see citing works
“During translation, GenAI can generate multiple outputs based on user prompts, which translators may use as drafts for further refinement (Hendy et al. 2023).”
Towards Effective Disambiguation for Machine Translation with Large Language Models
Iyer, V., Chen, P., & Birch, A. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.44 · Cited by 19 · see citing works
Teaching Large Language Models to Translate with Comparison
Zeng, J., Meng, F., Yin, Y., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29920 · Cited by 32 · see citing works
LLMCL-GEC: Advancing grammatical error correction with LLM-driven curriculum learning
Fang, T., Zhang, T., Wong, D. F., et al. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127397 · Cited by 13 · see citing works
Measuring translation trainees’ effort in AI-assisted post-editing: a multi-method approach
Yao, Y., Han, T., & Li, D. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2535239 · Cited by 20 · see citing works
“The GPT family, such as GPT-4, has been proven capable of achieving competitive translation quality compared to established neural MT models (Hendy et al. 2023; Jiao et al. 2023; Raunak, Menezes, et al. 2023).”
Weissweiler, L., Hofmann, V., Kantharuban, A., Cai, A., Dutt, R., Hengle, A., Kabra, A., Kulkarni, A., Vijayakumar, A., Yu, H., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.401 · Cited by 18 · see citing works
Traveling with ChatGPT and providing translation services for travelers
Kundan, Z., Hossain, M. S., Islam, M. S., Alam, M. N., & Parvez, M. O. (2024) · Tourism Review · doi:10.1108/tr-04-2024-0316 · Cited by 13 · see citing works
Algaraady, J., & Mahyoob, M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1526293 · Cited by 12 · see citing works
‘I am not a number’: on quantification and algorithmic norms in translation
Moorkens, J. (2023) · Perspectives · doi:10.1080/0907676x.2023.2278536 · Cited by 15 · see citing works
“Using similar processes but at a larger scale, generative tools using large language models have quickly reached similar levels of quality for well-supported languages, with reports of improved capabilities regarding context (Cas-tilho et al., 2023; Hendy et al., 2023).”
BackgroundStructuring Natural Language Requirements with Large Language Models
Norheim, J. J., & Rebentisch, E. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00013 · Cited by 11 · see citing works
“Current LLM can also, in many cases, achieve state-of-the-art machine translation [13], the task proposed in this paper.”
BackgroundDefinitions Matter: Guiding GPT for Multi-label Classification
Peskine, Y., Korenčić, D., Grubišić, I., Papotti, P., Raphaël Troncy, A., & Rosso, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.267 · Cited by 11 · see citing works
“…Er, 2023), summarization (Wang et al., 2023), question answering (Peinl and Wirth, 2023; Tan et al., 2023), information retrieval (Omar et al., 2023), machine translation (Hendy et al., 2023; Jiao et al., 2023), and inductive reasoning (Wei et al., 2022; Kojima et al., 2022; Zhong et al., 2023).”
Augmenting Large Language Model Translators via Translation Memories
Mu, Y., Reheman, A., Cao, Z., Fan, Y., Li, B., Li, Y., Xiao, T., Zhang, C., & Zhu, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.653 · Cited by 13 · see citing works
Abu-Rayyash, H. (2023) · CURRENT TRENDS IN TRANSLATION TEACHING AND LEARNING E · doi:10.51287/cttl20239 · Cited by 9 · see citing works
Speech translation for multilingual medical education leveraged by large language models
Iranzo-Sánchez, J., Santamaría-Jordà, J., Mas-Mollà, G., Díaz-Munío, G. V. G., Iranzo-Sánchez, J., Jorge, J., Silvestre-Cerdà, J. A., Giménez, A., Civera, J., Sanchís, A., et al. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103147 · Cited by 12 · see citing works
GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence
Wen, Z., Tian, Z., Wu, W., Yang, Y., Shi, Y., Huang, Z., & Li, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.262 · Cited by 15 · see citing works
Creative skills development: training translators to write in the era of AI
Arenas, A. G., & Asimakoulas, D. (2024) · HERMES - Journal of Language and Communication in Business · doi:10.7146/hjlcb.vi63.143078 · Cited by 9 · see citing works
GPT Struct Me: Probing GPT Models on Narrative Entity Extraction
Sousa, H., Guimar�ães, N., Jorge, A., & Campos, R. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00063 · Cited by 11 · see citing works
“Such advances have led several research studies to probe the capabilities of LLMs such as reasoning [9], semantics [10], machine translation [11], and named entity recognition [12], presenting state-of-the-art results in some of them.”
BackgroundLanguage model collaboration for relation extraction from classical Chinese historical documents
Tang, X., Wang, L., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104286 · Cited by 10 · see citing works
Cai, Y., & Tian, S. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2543209 · Cited by 10 · see citing works
“At the process level, studies have been undertaken to examine how GenAI supports specific stages such as source text comprehension, terminology management, context interpretation, MT, post-editing, and quality assurance Siu (2023); Hendy et al. (2023).”
On compositional generalization of transformer-based neural machine translation
Yin, Y., Fu, L., Li, Y., & Zhang, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102491 · Cited by 8 · see citing works
ChatGPT as a New Frontier in Arabic Education Technology
Sulaeman, I., Syuhadak, S., & Sulaeman, I. (2023) · Al-Arabi Jurnal Bahasa Arab dan Pengajarannya = Al-Arabi Journal of Teaching Arabic as a Foreign Language · doi:10.17977/um056v7i1p83-105 · Cited by 11 · see citing works
Wang, Z., Zhang, Y., Min, W., Guan, Q., & Yu, W. (2025) · Journal of Geography in Higher Education · doi:10.1080/03098265.2025.2549784 · Cited by 8 · see citing works
“LLMs exhibit excellent performance in traditional NLP tasks such as entity recognition (Wang et al., 2023; Zhou et al., 2023), language translation (Hendy et al., 2023; Jiao et al., 2023), and information extraction (Agrawal et al., 2022; Wei et al., 2023), as well as in specific domains like…”
Karakuş, İ. S., Strechen, I., Gupta, A., Nalaie, K., Chen, C. L., Hassett, L. C., & Barwise, A. K. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf150 · Cited by 8 · see citing works
Analysis of ChatGPT as a Question-Answering Tool
Pichappan, P., Krishnamurthy, M., & Vijayakumar, P. (2023) · Journal of Digital Information Management · doi:10.6025/jdim/2023/21/2/50-61 · Cited by 7 · see citing works
Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessment
Yang, H., Zhang, M., Tao, S., Wang, M., Wei, D., & Jiang, Y. (2024) · International Conference on Advanced Communications Technology (ICACT) · doi:10.23919/icact60172.2024.10471974 · Cited by 9 · see citing works
“In the domain of multilingual NLP tasks, (Jiao et al., 2023) (Hendy et al., 2023) have demonstrated that utilizing a prompt-based approach, instead of fine-tuning models, can enable LLMs to perform machine translation tasks.”
BackgroundA bibliometric analysis of literature published on ChatGPT and GPT
Safdar, M., Siddique, N., Gulzar, A., Adil, S. A., Yasin, H., & Khan, M. A. (2024) · Global Knowledge, Memory and Communication · doi:10.1108/gkmc-08-2023-0304 · Cited by 7 · see citing works
Aldosari, L. A., & Altuwairesh, N. (2025) · Perspectives · doi:10.1080/0907676x.2025.2464120 · Cited by 9 · see citing works
Luna, D. S., Broer, I., Bilandzić, H., Taddicken, M., Schuller, B. W., & Bürger, M. (2025) · Public Understanding of Science · doi:10.1177/09636625251328854 · Cited by 6 · see citing works
“Researchers have also explored the reliability of ComAI in specific fields like medicine (Johnson et al., 2023) and tasks like translation (Hendy et al., 2023).”
Conversational Ai: A Treatise About Vying Chatbots
Arya, S., Bhaskar, A., & Gupta, K. (2024) · International Conference on Disruptive Technologies (ICDT) · doi:10.1109/icdt61202.2024.10489545 · Cited by 6 · see citing works
“According to the study [18], the automated assessment measures used for the chatbots discussed, outputs at this time are insufficient.”
MethodHuman Evaluation for Translation Quality of ChatGPT: A Preliminary Study
Zhao, Y., Zhang, M., Chen, X., Deng, Y., Geng, A., Liu, L., Liu, X., Li, W., Jiang, Y., Yang, H., et al. (2023) · International Conference on Human-informed Translation and Interpreting Technology (HiT-IT) · doi:10.26615/issn.2683-0078.2023_023 · Cited by 6 · see citing works
“The result aligns with recent research in [7] and [2].”
BackgroundMethodResultBuilding accurate translation-tailored large language models with language-aware instruction tuning
Zan, C., Ding, L., Shen, L., Zhan, Y., Yang, X., & Liu, W. (2025) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2400458 · Cited by 6 · see citing works
- Authors
- Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., & Awadalla, H. H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.09210
- OpenAlex
- W4321472057
- Cited by
- 202
- References
- 36
- Citations by year
- 2026: 15 · 2025: 48 · 2024: 68 · 2023: 71
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Keywords
- GPT models, machine translation, ChatGPT, GPT-3.5, prompting strategies, domain shift robustness, document-level translation, low-resource languages, hybrid translation systems, human evaluation of machine translation, translation quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
