The DOI matches its Crossref record; 4 of the 34 references it cites were found in a registry or an index, and 30 are not checked yet.
Pretraining Language Models Using Translationese
Open accessAbstract
In this paper, we explore the utility of Translationese as synthetic data created using machine translation for pre-training language models (LMs) for low-resource languages (LRLs).Our simple methodology consists of translating large amounts of web-crawled monolingual documents (clean) into the LRLs, followed by filtering the translated documents using tiny LMs trained on small but clean LRL data.Taking the case of Indian languages, we pre-train LMs from scratch with 28M and 85M parameters, and then fine-tune them for 5 downstream natural language understanding (NLU) and 4 generative (NLG) tasks.We observe that pretraining on filtered synthetic data leads to relative performance drops of only 0.87% for NLU and 2.35% for NLG, compared to pre-training on clean data, and this gap further diminishes upon the inclusion of a small amount of clean data.We also study the impact of synthetic data filtering and the choice of source language for synthetic data generation.Furthermore, evaluating continually pre-trained larger models like Gemma-2B and Llama-3-8B in few-shot settings, we observe that using synthetic data is competitive with using clean data.Our findings suggest that synthetic data shows promise for bridging the pre-training gap between English and LRLs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Translation-based synthesis is an effective strategy for non-English languages, where abundant English resources are translated into the target language.
“For non-English languages, translation-based synthesis serves as an effective strategy, where abundant English resources are translated into the target language (Wang et al., 2025a; Doshi et al., 2024).”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Doshi et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Understanding Back-Translation at Scale
Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1045 · Cited by 1,082
XNLI: Evaluating Cross-lingual Sentence Representations
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1269 · Cited by 1,052
COMET: A Neural Framework for MT Evaluation
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.213 · Cited by 1,019
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Sequence-Level Knowledge Distillation
Kim, Y., & Rush, A. M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1139 · Cited by 836
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562
Kakwani, D., Kunchukuttan, A., Golla, S., Gokul Narayanan, C., Bhattacharyya, A., Khapra, M. M., & Kumar, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.445 · Cited by 501
A Survey of Multilingual Neural Machine Translation
Dabre, R., Chu, C., & Kunchukuttan, A. (2020) · ACM Computing Surveys · doi:10.1145/3406095 · Cited by 356
Self-citationImproving Neural Machine Translation Models with Monolingual Data
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1009 · Cited by 296
chrF++: words helping character n-grams
Popović, M. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4770 · Cited by 286
Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures
Suárez, P. O., Sagot, B., & Romary, L. (2019) · Publication Server of the Institute for German Language (Institute for German Language) · doi:10.14618/ids-pub-9021 · Cited by 282
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Hasan, T., Bhattacharjee, A., Islam, M. S., Mubasshir, K. S., Li, Y.-F., Kang, Y., Rahman, M. S., & Shahriyar, R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.413 · Cited by 280
Samanantar : The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
Ramesh, G., Doddapaneni, S., Bheemaraj, A., Jobanputra, M., Raghavan, A. K., Sharma, A., Sahoo, S., Diddee, H., Mahalakshmi, J., Kakwani, D., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00452 · Cited by 234
The Fact Extraction and VERification (FEVER) Shared Task
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5501 · Cited by 212
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation
Toral, A., Castilho, S., Hu, K., & Way, A. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6312 · Cited by 197
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset
Chen, Y., Liu, Y., Liang, C., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.449 · Cited by 183
Few-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 171
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/579or68a4ybs · Cited by 150
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
Kudugunta, S., Caswell, I. R., Zhang, B., Garcia, X., Xin, D., Kusupati, A., Stella, R., Bapna, A., & Firat, O. (2023) · neural information processing systems · doi:10.52202/075280-2940 · Cited by 125
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74
IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages
Kumar, A., Shrotriya, H., Sahu, P., Mishra, A., Dabre, R., Puduppully, R., Kunchukuttan, A., Khapra, M. M., & Kumar, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.360 · Cited by 47
Self-citationIndicXNLI: Evaluating Multilingual Inference for Indian Languages
Aggarwal, D., Gupta, V., & Kunchukuttan, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.755 · Cited by 38
Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users
Madhani, Y., Parthan, S., Bedekar, P., Gokul, N. C., Khapra, R., Kunchukuttan, A., Kumar, P., & Khapra, M. M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.4 · Cited by 32
Native Language Identification with User Generated Content
Goldin, G., Rabinovich, E., & Wintner, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1395 · Cited by 24
SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics
Deutsch, D., & Roth, D. (2020) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2020.nlposs-1.17 · Cited by 19
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
Qin, Y., Yuan, W., Neubig, G., & Liu, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1014 · Cited by 16
Ni, J., Jin, Z., Freitag, M., Sachan, M., & Schölkopf, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.389 · Cited by 7
Synergy with Translation Artifacts for Training and Inference in Multilingual Tasks
Oh, J., Ko, J., & Yun, S.-Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.452 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 23
- Nikolay Bogoychev and Rico Sennrich. 2019. Domain, translationese and noise in synthetic data for neural machine translation. CoRR, abs/1911.03362. · arXiv
- Ondrej Bojar, Vojtech Diatka, Pavel Rychlỳ, Pavel Stranák, Vít Suchomel, Ales Tamchyna, and Daniel Zeman. 2014. Hindencorp-hindi-english and hindi- only corpus for machine translation. In LREC, pages 3550-3555.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building mono- lingual corpora, benchmark and models for indic languages. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402-12426.
- Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Pudup- pully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible ma- chine translation models for all 22 scheduled indian languages. Transactions on Machine Learning Re- search.
- Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 lan- guages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 759-765, Istanbul, Turkey. Euro- pean Language Resources Association (ELRA).
- Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation eval- uation. CoRR, abs/1906.09833. · arXiv
- Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland. Association for Com- putational Linguistics.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685. · arXiv
- Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, NC Gokul, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020a. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4948- 4961.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
- Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite mul- tilingual dataset. Advances in Neural Information Processing Systems, 35:31809-31826.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
- Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged back-translation revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5990-5997, Online. Association for Computational Linguistics. · DOI
- Anthony McEnery, Paul Baker, Robert Gaizauskas, and Hamish Cunningham. 2000. Emille: Building a cor- pus of south asian languages. In Proceedings of the International Conference on Machine Translation and Multilingual Applications in the new Millennium: MT 2000.
- Jihyung Moon, Hyunchang Cho, and Eunjeong L. Park. 2020. Revisiting round-trip translation for quality estimation. In Proceedings of the 22nd Annual Con- ference of the European Association for Machine Translation, pages 91-104, Lisboa, Portugal. Euro- pean Association for Machine Translation.
- Maja Popović, Alberto Poncelas, Marija Brkic, and Andy Way. 2020. Neural machine translation for translating into Croatian and Serbian. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 102-113, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics (ICCL).
- Ella Rabinovich and Shuly Wintner. 2015. Unsuper- vised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419- 432.
- Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Pro- ceedings of the Sixth Conference on Machine Trans- lation, pages 1030-1040, Online. Association for Computational Linguistics.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
- Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing
Cited by
Sorted by citations
Scaling Data-Constrained Language Models with Synthetic Data
Kiyomaru, H., Oda, Y., Kodama, T., Liu, C., & Kawahara, D. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.52
How Can Synthetic Data Improve Multilingual Language Model Pretraining? A Data Quality Perspective
Zhu, T., Liu, Q., Ma, C., Zhang, J., Dou, L., He, J., & Chen, S. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1002
Metadata
- Authors
- Doshi, M., Dabre, R., & Bhattacharyya, P.
- Affiliations
- Indian Institute of Technology Bombay (IN)
- Indian Institute of Technology Madras (IN)
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 5843-5862
- OpenAlex
- W4404782570
- Cited by
- 1
- References
- 4
- Citations by year
- 2026: 1
- FWCI
- 0.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- translationese, synthetic data, low-resource languages, machine translation, Indian languages, natural language understanding, natural language generation, few-shot learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
