From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
Open accessAbstract
In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language.The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation.We performed our experiments with three languages, each using a non-Latin script-Ukrainian, Arabic, and Georgian.Our approach demonstrates improved language performance while reducing computational costs.It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar.Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Miquel Esplà, Mikel Forcada, Gema Ramírez-Sánchez, and Hieu Hoang. 2019. ParaCrawl: Web-scale paral- lel corpora for the languages of the EU. In Proceed- ings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 118-119, Dublin, Ire- land. European Association for Machine Translation.
- Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural net- works. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings. "hemanth kumar". Tamil-mistral-7b-v0.
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning ai with shared human values. Pro- ceedings of the International Conference on Learning Representations (ICLR).
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021b. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
- Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Ara- bic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Lin- guistics.
- Mustafa Jarrar, Fadi A. Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Wählisch. 2023. Lisan: Yemeni, iraqi, libyan, and sudanese arabic dialect corpora with morphological annotations. In 20th ACS/IEEE International Conference on Com- puter Systems and Applications, AICCSA 2023, Giza, Egypt, December 4-7, 2023, pages 1-7. IEEE. · DOI
- Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. GigaBERT: Zero-shot transfer learning from English to Arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP).
- Leah S. Larkey, Lisa Ballesteros, and Margaret E. Con- nell. 2002. Improving stemming for arabic infor- mation retrieval: light stemming and co-occurrence analysis. In Proceedings of the 25th Annual Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '02, page 275-282, New York, NY, USA. Association for Computing Machinery. · DOI
- Kelly Marchisio, Wei-Yin Ko, Alexandre Bérard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in llms. · DOI
- Mohamed Megahed. 2021. Sequence labeling archi- tectures in diglossia -a case study of arabic and its dialects. · DOI
- Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021b. How good is your to- kenizer? on the monolingual performance of multi- lingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics. · DOI · Paper page
- Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring design choices for building language- specific llms. · DOI
- Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. 2023. SkyPilot: An in- tercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23), pages 437-455, Boston, MA. USENIX Association.
Cited by
Sorted by citations
Framing the Language: Fine-Tuning Gemma 3 for Manipulation Detection
Khandoga, M., Kostiuk, Y., Polishko, A., Kozlov, K., Filipchuk, Y., & Kiulian, A. (2025) · Ukrainian Natural Language Processing Workshop (UNLP) · doi:10.18653/v1/2025.unlp-1.6 · Cited by 4 · see citing works
Self-citationPlatzgummer, V., McCrae, J. P., & Ahmadi, S. (2026) · Language Policy · doi:10.1007/s10993-026-09811-8 · Cited by 3 · see citing works
Rodríguez, P., Suárez, S., Gamallo, P., & Docío, S. S. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.240 · Cited by 1 · see citing works
Advancing Tigrinya Multiclass Text Classification with Pre-trained Models: A Comparative AnalysisPreprint
Tesfai, D., Ning, H., Sarwatt, D. S., & Ding, J. (2025) · doi:10.36227/techrxiv.175322819.98359474/v1 · Cited by 1 · see citing works
Writing-System-Level Tokenizer Adaptation for Byte-Level BPE
Didenko, B. (2026) · arXiv · doi:10.48550/arxiv.2608.00582
R-BPE: Improving BPE-Tokenizers with Token Reuse
Hamdan, N., Mraikhat, O. R. A., & Zaraket, F. A. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1169
Self-citationIt's All Connected: A Survey for Multimodal Arabic AIPreprint
Aldabbas, F., Elsafty, H., & Sifa, R. (2025) · Research Square · doi:10.21203/rs.3.rs-8007923/v1
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models
Alqahtani, S., Nayeem, M. T., Laskar, M. T. R., Mohiuddin, T., & Bari, M. S. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.394
- Authors
- Kiulian, A., Polishko, A., Khandoga, M., Kostiuk, Y., Gabrielli, G., Gagała, Ł., Zaraket, F., Abu Obaida, Q., Garud, H., Wing Yee Mak, W., Chaplynskyi, D., Amor, S., & Peradze, G.
- Affiliations
- University of Dundee (GB)
- Doha Institute for Graduate Studies (QA)
- Google (United States) (US)
- Ukrainian Catholic University (UA)
- Venue
- Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025)
- Published
- 2025
- Pages
- 1-13
- OpenAlex
- W4412944228
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 1.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Ukrainian Natural Language Processing Workshop (UNLP) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- large language models, underrepresented languages, low-resource languages, bilingual language models, multilingual NLP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
