Automatic Classification of Human Translation and Machine Translation: A Study from the Perspective of Lexical Diversity
Open accessAbstract
By using a trigram model and fine-tuning a pretrained BERT model for sequence classification, we show that machine translation and human translation can be classified with an accuracy above chance level, which suggests that machine translation and human translation are different in a systematic way. The classification accuracy of machine translation is much higher than of human translation. We show that this may be explained by the difference in lexical diversity between machine translation and human translation. If machine translation has independent patterns from human translation, automatic metrics which measure the deviation of machine translation from human translation may conflate difference with quality. Our experiment with two different types of automatic metrics shows correlation with the result of the classification task. Therefore, we suggest the difference in lexical diversity between machine translation and human translation be given more attention in machine translation evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1× · see citing works
“Second, we choose words whose vectors are close to the rare words from the pretrained GloVe embeddings (Pennington et al., 2014), which are computationally less expensive than contextualized word embeddings like BERT.”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1× · see citing works
“The assumption is that the closer a machine translation is to a professional human translation, the better it is (Papineni et al., 2002).”
BackgroundGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1× · see citing works
“In recent years, progress has been made in MT research and development, and it is claimed that MT achieves human parity in some tasks (Wu et al., 2016; Hassan et al., 2018; Popel et al., 2020).”
BackgroundBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 1× · see citing works
“The majority of automatic MT metrics developed in recent years such as BERTScore (Zhang et al., 2019) and Yisi (Lo, 2019) adopt contextualized embeddings.”
BackgroundParallel Data, Tools and Interfaces in OPUS
Tiedemann, J. (2012) · Language Resources and Evaluation · Cited by 1,421 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,054 · see citing works
McCarthy, P. M., & Jarvis, S. (2010) · Behavior Research Methods · doi:10.3758/brm.42.2.381 · Cited by 905 · cited 2× · see citing works
“To compute the lexical diversity of the texts, based on research by McCarthy and Jarvis (2010) and Vanmassenhove et al. (2019), we choose the measure of textual lexical diversity (MTLD) (McCarthy, 2005), which is reasonably robust to text length difference.”
MethodCOMET: A Neural Framework for MT Evaluation
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.213 · Cited by 767 · see citing works
Achieving Human Parity on Automatic Chinese to English News TranslationPreprint
Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X.-D., Junczys-Dowmunt, M., Lewis, W., Li, M., et al. (2018) · arXiv.org · doi:10.48550/arxiv.1803.05567 · Cited by 652 · cited 1× · see citing works
“In recent years, progress has been made in MT research and development, and it is claimed that MT achieves human parity in some tasks (Wu et al., 2016; Hassan et al., 2018; Popel et al., 2020).”
BackgroundCorpus Linguistics and Translation Studies*
Baker, M. (2019) · Routledge eBooks · doi:10.4324/9780429024221-2 · Cited by 602 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · cited 5× · see citing works
“The MoverScore result in Table 8 shows a different pattern from the BLEU scores.”
MethodInfluentialCross-Linguistic Variation in System and Text
Teich, E. (2003) · De Gruyter eBooks · doi:10.1515/9783110896541 · Cited by 352 · cited 1× · see citing works
“Mean-while, computational approaches (Teich, 2003; Volansky et al., 2015) contribute evidence for some translation universals.”
BackgroundPopel, M., Tomkova, M., Tomek, J., Kaiser, Ł., Uszkoreit, J., Bojar, O., & Žabokrtský, Z. (2020) · Nature Communications · doi:10.1038/s41467-020-18073-9 · Cited by 346 · cited 1× · see citing works
“In recent years, progress has been made in MT research and development, and it is claimed that MT achieves human parity in some tasks (Wu et al., 2016; Hassan et al., 2018; Popel et al., 2020).”
BackgroundHas Machine Translation Achieved Human Parity? A Case for Document-level Evaluation
Läubli, S., Sennrich, R., & Volk, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1512 · Cited by 294 · cited 1× · see citing works
“However, these statements are challenged by other researchers and remain open to debate (L¨aubli et al., 2018; Toral et al., 2018; Toral, 2020).”
BackgroundOn the features of translationese
Volansky, V., Ordan, N., & Wintner, S. (2015) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqt031 · Cited by 223 · cited 2× · see citing works
“Meanwhile, computational approaches (Teich, 2003; Volansky et al., 2015) contribute evidence for some translation universals.”
BackgroundAttaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation
Toral, A., Castilho, S., Hu, K., & Way, A. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6312 · Cited by 197 · cited 2× · see citing works
“However, these statements are challenged by other researchers and remain open to debate (Läubli et al., 2018; Toral et al., 2018; Toral, 2020).”
BackgroundA survey of machine translation: its history, current status, and future prospects
Slocum, J. (1985) · OLAC - Open Language Archives Community · Cited by 142 · see citing works
Lo, C.-K. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5358 · Cited by 114 · cited 2× · see citing works
“The majority of automatic MT metrics developed in recent years such as BERTScore (Zhang et al., 2019) and Yisi (Lo, 2019) adopt contextualized embeddings.”
BackgroundLanguage Models for Machine Translation: Original vs. Translated Texts
Lembersky, G., Ordan, N., & Wintner, S. (2012) · Computational Linguistics · doi:10.1162/coli_a_00111 · Cited by 100 · cited 1× · see citing works
“Automatic means to distinguish translated texts and originally written texts have been developed and generally achieve high accuracy (Baroni and Bernardini, 2006; Ilisei et al., 2010; Lembersky et al., 2012; Rabinovich and Wintner, 2015).”
BackgroundMachine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation
Vanmassenhove, E., Shterionov, D., & Gwilliam, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.188 · Cited by 99 · see citing works
Identification of Translationese: A Machine Learning Approach
Ilisei, I., Inkpen, D. Z., Pastor, G. C., & Mitkov, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12116-6_43 · Cited by 74 · cited 2× · see citing works
“Automatic means to distinguish translated texts and originally written texts have been developed and generally achieve high accuracy (Baroni and Bernardini, 2006; Ilisei et al., 2010; Lembersky et al., 2012; Rabinovich and Wintner, 2015).”
BackgroundLost in Translation: Loss and Decay of Linguistic Richness in Machine TranslationPreprint
Vanmassenhove, E., Shterionov, D., & Way, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.12068 · Cited by 74 · cited 7× · see citing works
“Different from the conventional practice of MT evaluation that treats HT as the gold standard, some studies adopt a descriptive approach to comparing MT and HT (Bizzoni et al., 2020; Ahrenberg, 2017; Vanmassenhove et al., 2019).”
BackgroundMethodResultInfluentialPost-editese: an Exacerbated TranslationesePreprint
Toral, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00900 · Cited by 70 · cited 3× · see citing works
“There are some studies that compare translation varieties such as professional and student translations and post-edited MT (Kunilovskaya and Lapshinova-Koltunski, 2019; Toral, 2019; Popović, 2020).”
BackgroundMethodUnsupervised Identification of Translationese
Rabinovich, E., & Wintner, S. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00148 · Cited by 51 · cited 1× · see citing works
“Automatic means to distinguish translated texts and originally written texts have been developed and generally achieve high accuracy (Baroni and Bernardini, 2006; Ilisei et al., 2010; Lembersky et al., 2012; Rabinovich and Wintner, 2015).”
BackgroundComparing Machine Translation and Human Translation: A Case Study
Ahrenberg, L. (2017) · doi:10.26615/978-954-452-042-7_003 · Cited by 35 · cited 1× · see citing works
“Different from the conventional practice of MT evaluation that treats HT as the gold standard, some studies adopt a descriptive approach to comparing MT and HT (Bizzoni et al., 2020; Ahrenberg, 2017; Vanmassenhove et al., 2019).”
MethodHow Human is Machine Translationese? Comparing Human and Machine Translations of Text and Speech
Bizzoni, Y., Juzek, T. S., España-Bonet, C., Dutta Chowdhury, K., van Genabith, J., & Teich, E. (2020) · Proceedings of the 17th International Conference on Spoken Language Translation · doi:10.18653/v1/2020.iwslt-1.34 · Cited by 34 · cited 4× · see citing works
“Among these studies, Bizzoni et al. (2020) find that MT shows independent patterns of transla-tionese and it resembles HT only partly.”
BackgroundMethodResultInfluentialRubino, R., Lapshinova‐Koltunski, E., & van Genabith, J. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1110 · Cited by 28 · cited 2× · see citing works
“Rubino et al. (2016) perform the classification between originally written texts and translations as well as between professional and student translations.”
BackgroundMethodReassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019Preprint
Toral, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05738 · Cited by 24 · cited 1× · see citing works
“However, these statements are challenged by other researchers and remain open to debate (Läubli et al., 2018; Toral et al., 2018; Toral, 2020).”
BackgroundTranslationese Features as Indicators of Quality in English-Russian Human Translation
Kunilovskaya, M., & Lapshinova‐Koltunski, E. (2019) · doi:10.26615/issn.2683-0078.2019_006 · Cited by 12 · see citing works
On the differences between human translations
Popović, M. (2020) · Dublin City University Open Access Institutional Repository (Dublin City University) · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 31 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Matsui, K. (2025) · Perspectives on Medical Education · doi:10.5334/pme.1929 · Cited by 6 · see citing works
Shi, L. M. (2023) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.3834 · Cited by 3 · see citing works
“Machine translation systems are frequently used for online and machine translation[16].”
Background
- Authors
- Fu, Y., & Nederhof, M.-J.
- Affiliations
- University of St Andrews (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2105.04616
- OpenAlex
- W3162850041
- Cited by
- 1
- References
- 31
- Citations by year
- 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- St Andrews Research Repository (St Andrews Research Repository) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Translation Studies and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- machine translation, human translation, lexical diversity, trigram language model, BERT fine-tuning, automatic evaluation metrics, machine translation evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
