A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
Open accessAbstract
We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages.We then compare the performance of OSCARbased and Wikipedia-based ELMo embeddings for these languages on the part-ofspeech tagging and parsing tasks.We show that, despite the noise in the Common-Crawlbased OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia.They actually equal or improve the current state of the art in tagging and parsing for all five languages.In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the crosslingual benefit of multilingual embedding architectures.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
38% of ROOTS consists of text extracted from OSCAR, filtered with the help of native speakers.
“38% consists of text extracted from a pre-processed web crawl, OSCAR (Ortiz Suárez et al. (2020)), filtered with the help of native speakers”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Ortiz Suárez et al. (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 24
- Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1638-1649. Association for Computa- tional Linguistics.
- Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations 22 http://prairie-institute.fr/ 23 https://wiki.inria.fr/wikis/ ClustersSophia for multilingual NLP. In Proceedings of the Seven- teenth Conference on Computational Natural Lan- guage Learning, pages 183-192, Sofia, Bulgaria. Association for Computational Linguistics.
- Bernd Bohnet, Ryan McDonald, Gonçalo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta- BiLSTM model over context sensitive token encod- ings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2642-2652, Melbourne, Australia. Association for Computational Linguis- tics.
- Matthias Buch-Kromann. 2003. The danish depen- dency treebank and the dtag treebank tool. In 2nd Workshop on Treebanks and Linguistic Theories (TLT), Sweden, pages 217-220.
- Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55-64, Brussels, Belgium. Association for Compu- tational Linguistics.
- Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
- Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar- mand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the 11th Language Resources and Evaluation Con- ference, Miyazaki, Japan. European Language Re- source Association.
- Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651. · arXiv
- Languages, 1 Model: Parsing Universal De- pendencies Universally. arXiv e-prints, page arXiv:1904.02099. · arXiv
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv e- prints, page arXiv:1909.11942. · arXiv
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining ap- proach. CoRR, abs/1907.11692. · arXiv
- Louis Martin, Benjamin Muller, Pedro Javier Or- tiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. CamemBERT: a Tasty French Language Model. arXiv e-prints, page arXiv:1911.03894. · arXiv
- Rada Mihalcea. 2007. Using Wikipedia for automatic word sense disambiguation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computa- tional Linguistics; Proceedings of the Main Confer- ence, pages 196-203, Rochester, New York. Associ- ation for Computational Linguistics.
- Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Ad- vances in pre-training distributed word representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018.
- Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2012. A universal part-of-speech tagset. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation, LREC 2012, Istan- bul, Turkey, May 23-25, 2012, pages 2089-2096. Eu- ropean Language Resources Association (ELRA).
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Explor- ing the Limits of Transfer Learning with a Uni- fied Text-to-Text Transformer. arXiv e-prints, page arXiv:1910.10683. · arXiv
- Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal dependency pars- ing with multi-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Pars- ing from Raw Text to Universal Dependencies, pages 113-123, Brussels, Belgium. Association for Com- putational Linguistics.
- Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197-207, Brussels, Belgium. Association for Computational Linguistics. · DOI
- Milan Straka, Jana Straková, and Jan Hajic. 2019. Eval- uating contextualized embeddings on 54 languages in POS tagging, lemmatization and dependency pars- ing. CoRR, abs/1908.07448. · arXiv
- Mariona Taulé, Maria Antònia Martí, and Marta Re- casens. 2008. Ancora: Multilevel annotated corpora for catalan and spanish. In Proceedings of the In- ternational Conference on Language Resources and Evaluation, LREC 2008, 26 May -1 June 2008, Mar- rakech, Morocco. European Language Resources Association.
- Trieu H. Trinh and Quoc V. Le. 2018. A sim- ple method for commonsense reasoning. CoRR, abs/1806.02847. · arXiv
- Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2019. CC- Net: Extracting High Quality Monolingual Datasets from Web Crawl Data. arXiv e-prints, page arXiv:1911.00359. · arXiv
- Fei Wu and Daniel S. Weld. 2010. Open information extraction using Wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Compu- tational Linguistics, pages 118-127, Uppsala, Swe- den. Association for Computational Linguistics.
- Daniel Zeman, Jan Hajič, Martin Popel, Martin Pot- thast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Mul- tilingual parsing from raw text to universal depen- dencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Univer- sal Dependencies, pages 1-21, Brussels, Belgium. Association for Computational Linguistics.
Cited by
Sorted by citations
KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media
Safaya, A., Abdullatif, M., & Yüret, D. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.271 · Cited by 313 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsPreprint
Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06678 · Cited by 200 · see citing works
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00447 · Cited by 179 · see citing works
Self-citationPre-Training BERT on Arabic Tweets: Practical ConsiderationsPreprint
Abdelalí, A., Hassan, S., Mubarak, H., Darwish, K. M., & Samih, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10684 · Cited by 134 · see citing works
Ogueji, K., Zhu, Y., & Lin, J. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.11 · Cited by 124 · see citing works
AraELECTRA: Pre-Training Text Discriminators for Arabic Language UnderstandingPreprint
Antoun, W., Baly, F., & Hajj, H. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15516 · Cited by 120 · see citing works
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97 · see citing works
Müller, B., Anastasopoulos, A., Sagot, B., & Seddah, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.38 · Cited by 96 · see citing works
Self-citationAraGPT2: Pre-Trained Transformer for Arabic Language GenerationPreprint
Antoun, W., Baly, F., & Hajj, H. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15520 · Cited by 92 · see citing works
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/579or68a4ybs · Cited by 87 · see citing works
Self-citationCoVoST 2 and Massively Multilingual Speech-to-Text TranslationPreprint
Wang, C., Wu, A., & Pino, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.10310 · Cited by 82 · see citing works
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
Xu, H., Durme, B. V., & Murray, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.534 · Cited by 68 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
medBERT.de: A comprehensive German BERT model for the medical domain
Bressem, K. K., Papaioannou, J.-M., Grundmann, P., Borchert, F., Adams, L. C., Liu, L., Busch, F., Xu, L., Loyen, J. P., Niehues, S. M., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121598 · Cited by 60 · see citing works
HeBERT and HebEMO: A Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition
Chriqui, A., & Yahav, I. (2022) · INFORMS Journal on Data Science · doi:10.1287/ijds.2022.0016 · Cited by 52 · see citing works
TurkishBERTweet: Fast and reliable large language model for social media analysis
Najafi, A., & Varol, O. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124737 · Cited by 48 · see citing works
Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go
Adebara, I., & Abdul-Mageed, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.265 · Cited by 45 · see citing works
Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection
Farha, I. A., & Magdy, W. (2021) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 45 · see citing works
Validating daily social media macroscopes of emotions
Pellert, M., Metzler, H., Matzenberger, M., & García, D. (2022) · Scientific Reports · doi:10.1038/s41598-022-14579-y · Cited by 45 · see citing works
Indic-Transformers: An Analysis of Transformer Language Models for Indian LanguagesPreprint
Jain, K., Deshpande, A., Shridhar, K., Laumann, F., & Dash, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02323 · Cited by 39 · see citing works
Pabón, O. S., Montenegro, O., Torrente, M., Rodríguez‐González, A., Provencio, M., & Ernestina, M. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.913 · Cited by 36 · see citing works
AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level
Seker, A., Bandel, E., Bareket, D., Brusilovsky, I., Greenfeld, R. S., & Tsarfaty, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.4 · Cited by 35 · see citing works
Resources for Turkish natural language processing: A critical survey
Çöltekin, Ç., Doğruöz, A. S., & Çetinoğlu, Ö. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09605-4 · Cited by 34 · see citing works
Machine Translation into Low-resource Language Varieties
Kumar, S., Anastasopoulos, A., Wintner, S., & Tsvetkov, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.16 · Cited by 29 · see citing works
Predicting the Performance of Multilingual NLP ModelsPreprint
Srinivasan, A., Sitaram, S., Ganu, T., Dandapat, S., Bali, K., & Choudhury, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08875 · Cited by 28 · see citing works
IndoNLI: A Natural Language Inference Dataset for Indonesian
Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.821 · Cited by 27 · see citing works
Tucano: Advancing neural text generation for Portuguese
Corrêa, N. K., Sen, A., Falk, S., & Fatimah, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101325 · Cited by 25 · see citing works
IndicXNLI: Evaluating Multilingual Inference for Indian Languages
Aggarwal, D., Gupta, V., & Kunchukuttan, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.755 · Cited by 25 · see citing works
Distributed Deep Learning in Open CollaborationsPreprint
Diskin, M., Bukhtiyarov, A., Ryabinin, M., Saulnier, L., Lhoest, Q., Sinitsin, A., Popov, D., Pyrkin, D., Kashirin, M., Borzunov, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10207 · Cited by 25 · see citing works
Yarmohammadi, M., Wu, S., Marone, M., Xu, H., Ebner, S., Qin, G., Chen, Y., Guo, J., Harman, C., Murray, K., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.149 · Cited by 23 · see citing works
TeenyTinyLlama: Open-source tiny language models trained in Brazilian Portuguese
Corrêa, N. K., Falk, S., Fatimah, S., Sen, A., & Oliveira, N. F. D. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100558 · Cited by 22 · see citing works
Williams, E., Rodrigues, P., & Novak, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02431 · Cited by 22 · see citing works
Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2021) · HAL · doi:10.14618/ids-pub-10468 · Cited by 21 · see citing works
Self-citationAraDepSu: Detecting Depression and Suicidal Ideation in Arabic Tweets Using Transformers
Hassib, M., Hossam, N., Sameh, J., & Torki, M. (2022) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/2022.wanlp-1.28 · Cited by 21 · see citing works
AfroLID: A Neural Language Identification Tool for African Languages
Adebara, I., Elmadany, A., Abdul-Mageed, M., & Inciarte, A. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.128 · Cited by 20 · see citing works
Detect Arabic fake news through deep learning models and Transformers
Alnabrisi, I. K., & Saad, M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123997 · Cited by 20 · see citing works
A benchmark for evaluating Arabic contextualized word embedding models
Elnagar, A., Yagi, S., Mansour, Y., Lulu, L., & Fareh, S. I. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103452 · Cited by 19 · see citing works
Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
de Vries, W., Bartelds, M., Nissim, M., & Wieling, M. B. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.433 · Cited by 19 · see citing works
Vázquez, A., Zorrilla, A. L., Olaso, J. M., & Torres, M. I. (2023) · Sensors · doi:10.3390/s23031423 · Cited by 19 · see citing works
A Large and Diverse Arabic Corpus for Language Modeling
Ali, A. R., Siddiqui, M. A., Algunaibet, R., & Ali, H. R. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.09.086 · Cited by 18 · see citing works
On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
Chiang, C.-H., & Lee, H.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21295 · Cited by 18 · see citing works
PersianLLaMA: Towards Building First Persian Large Language ModelPreprint
Abbasi, M. A., Ghafouri, A., Firouzmandi, M., Naderi, H., & Bidgoli, B. M. (2023) · Research Square · doi:10.21203/rs.3.rs-3789059/v1 · Cited by 18 · see citing works
Ahuir, V., Hurtado, L.-F., González, J. Á., & Segarra, E. (2021) · Applied Sciences · doi:10.3390/app11219872 · Cited by 17 · see citing works
Yamagishi, Y., Nakamura, Y., Hanaoka, S., & Abe, O. (2025) · JMIR Cancer · doi:10.2196/57275 · Cited by 16 · see citing works
The ROOTS Search Tool: Data Transparency for LLMs
Piktus, A., Akiki, C., Villegas, P., Laurençon, H., Dupont, G., Luccioni, A. S., Jernite, Y., & Rogers, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.29 · Cited by 16 · see citing works
From Translation to Generative LLMs: Classification of Code-Mixed Affective Tasks
Yadav, A., Garg, T., Klemen, M., Ulčar, M., Agarwal, B., & Robnik‐Šikonja, M. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2025.3553399 · Cited by 15 · see citing works
Kao, W.-T., & Lee, H.-Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.189 · Cited by 15 · see citing works
MFAQ: a Multilingual FAQ Dataset
Bruyn, M. D., Lotfi, E., Buhmann, J., & Daelemans, W. M. P. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.1 · Cited by 15 · see citing works
Bertinho: Galician BERT RepresentationsPreprint
Vilares, D., García, M., & Gómez‐Rodríguez, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13799 · Cited by 14 · see citing works
Ziyaden, A., Yelenov, A., Hajiyev, F., Rustamov, S., & Pak, A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1974 · Cited by 13 · see citing works
Dense Image Captioning in Hindi
Gill, K., Saha, S., & Mishra, S. K. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9658586 · Cited by 13 · see citing works
Continual Learning Under Language Shift
Gogoulou, E., Lesort, T., Boman, M., & Nivre, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70563-2_6 · Cited by 13 · see citing works
TigerLLM - A Family of Bangla Large Language Models
Raihan, N., & Zampieri, M. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-short.69 · Cited by 12 · see citing works
Named Entity Recognition for Low-Resource Languages - Profiting from Language Families
Torge, S., Politov, A., Lehmann, C. U., Saffar, B., & Tao, Z. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.1 · Cited by 11 · see citing works
PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Reid, M., & Artetxe, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.58 · Cited by 11 · see citing works
A Multilingual Neural Coaching Model with Enhanced Long-term Dialogue Structure
Zorrilla, A. L., & Torres, M. I. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3487066 · Cited by 11 · see citing works
Idrissi-Yaghir, A., Schäfer, H., Bauer, N., & Friedrich, C. M. (2023) · SN Computer Science · doi:10.1007/s42979-022-01563-6 · Cited by 11 · see citing works
Generating Intelligible Plumitifs Descriptions: Use Case Application with Ethical Considerations
Beauchemin, D., Garneau, N., Gaumond, E., Déziel, P.-L., Khoury, R., & Lamontagne, L. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.3 · Cited by 11 · see citing works
Alrayzah, A., Alsolami, F. J., & Saleh, M. (2024) · Applied Sciences · doi:10.3390/app14125294 · Cited by 10 · see citing works
FairDistillation: Mitigating Stereotyping in Language Models
Delobelle, P., & Berendt, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26390-3_37 · Cited by 10 · see citing works
Sitaram, S., Choudhury, M., Patra, B., Chaudhary, V., Ahuja, K., & Bali, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.3 · Cited by 10 · see citing works
Raychawdhary, N., Das, A. K., Dozier, G. V., & Seals, C. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.208 · Cited by 10 · see citing works
On the Usability of Transformers-based models for a French Question-Answering task
QWANT, 9. N.-S.-S., Cattan, O., Université Paris-Saclay, C., Servan, C., QWANT, 9. N.-S.-S., Rosset, S., & Université Paris-Saclay, C. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_029 · Cited by 9 · see citing works
Pre-trained Language Models for Tagalog with Multi-source Data
Jiang, S., Fu, Y., Lin, X., & Lin, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_17 · Cited by 9 · see citing works
Laippala, V., Rönnqvist, S., Oinonen, M., Kyröläinen, A.-J., Salmela, A., Biber, D., Egbert, J., & Pyysalo, S. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09624-1 · Cited by 9 · see citing works
Gessler, L., & Zeldes, A. (2022) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2022.mrl-1.9 · Cited by 9 · see citing works
Improved Prioritization of Software Development Demands in Turkish With Deep Learning-Based NLP
Tunalı, V. (2022) · IEEE Access · doi:10.1109/access.2022.3167269 · Cited by 9 · see citing works
Pre-Training a Language Model Without Human LanguagePreprint
Chiang, C.-H., & Lee, H.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11995 · Cited by 9 · see citing works
Towards Detecting Political Bias in Hindi News Articles
Agrawal, S., Gupta, K., Gautam, D., & Mamidi, R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.17 · Cited by 8 · see citing works
Wada, T., Iwata, T., Matsumoto, Y., Baldwin, T., & Lau, J. H. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.2 · Cited by 8 · see citing works
The Large Language Model GreekLegalRoBERTa
Saketos, V., Pantazi, D.-A., & Koubarakis, M. (2024) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3688671.3688770 · Cited by 7 · see citing works
ParaShoot: A Hebrew Question Answering Dataset
Keren, O., & Levy, O. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.11 · Cited by 7 · see citing works
Transformer-based Hebrew NLP models for Short Answer Scoring in Biology
Schleifer, A. G., Klebanov, B. B., Ariely, M., & Alexandron, G. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.46 · Cited by 7 · see citing works
An Enhanced Transformer-Based Approach with Meta-Ensemble Learning for Arabic Sentiment Analysis
Ashraf, M., Marzouk, M., Kora, R., & Mohammed, A. (2023) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc58832.2023.10278312 · Cited by 7 · see citing works
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions
Köksal, A., Thaler, M., Imani, A., Üstün, A., Korhonen, A., & Schütze, H. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.18 · Cited by 7 · see citing works
Evaluation of Arabic-Based Contextualized Word Embedding Models
Yagi, S., Mansour, Y., Kamalov, F., & Elnagar, A. (2021) · International Conference on Asian Language Processing · doi:10.1109/ialp54817.2021.9675208 · Cited by 7 · see citing works
Analyzing the unrestricted web: The finnish corpus of online registers
Skantsi, V., & Laippala, V. (2023) · Nordic Journal of Linguistics · doi:10.1017/s0332586523000021 · Cited by 7 · see citing works
Alrowili, S., & Shanker, V. K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.108 · Cited by 7 · see citing works
Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language ModelsPreprint
Carrino, C. P., Armengol-Estapé, J., Bonet, O. D. G., Gutiérrez-Fandiño, A., González-Agirre, A., Krallinger, M., & Villegas, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07765 · Cited by 7 · see citing works
Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning
Bandari, A., Yin, L., Hsieh, C.-Y., Jaiswal, A. K., Chen, T., Shen, L., Krishna, R., & Liu, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1004 · Cited by 6 · see citing works
Multilingual Sequence-to-Sequence Models for Hebrew NLP
Eyal, M., Noga, H., Aharoni, R., Szpektor, I., & Tsarfaty, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.487 · Cited by 6 · see citing works
iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic
Henia, W., Rjab, O., Haddad, H., & Fourati, C. (2021) · Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda (NLP4IF) · doi:10.18653/v1/2021.nlp4if-1.17 · Cited by 6 · see citing works
Moses and the Character-Based Random Babbling Baseline: CoAStaL at AmericasNLP 2021 Shared Task
Bollmann, M., Aralikatte, R., Bello, H. R. M., Hershcovich, D., de Lhoneux, M., & Søgaard, A. (2021) · Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP) · doi:10.18653/v1/2021.americasnlp-1.28 · Cited by 6 · see citing works
Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations
Xie, Z., Zhao, H., Yu, T., & Ли, Ш. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.379 · Cited by 5 · see citing works
Predictive typing method for Persian office automation
Nouraei, B., Shanbehzadeh, J., & Asghari, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107792 · Cited by 5 · see citing works
Jouitteau, M. (2023) · Lapurdum · doi:10.4000/127u1 · Cited by 5 · see citing works
SnapSeek: A Multimodal Video Retrieval System with Context Awareness for AI Challenge 2024
Ho-Le, M.-Q., Ho, D.-K., Do-Huu, H.-H., Le-Hinh, N.-T., Vo-Hoang, H.-V., Ninh, V.-T., & Tran, M. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4291-5_23 · Cited by 5 · see citing works
NepaliBERT: Pre-training of Masked Language Model in Nepali Corpus
Pudasaini, S., Shakya, S., Tamang, A., Adhikari, S., Thapa, S., & Lamichhane, S. (2023) · International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC) · doi:10.1109/i-smac58438.2023.10290690 · Cited by 5 · see citing works
Quantifying Gender Bias in Arabic Pre-Trained Language Models
Alrajhi, W., Al-Khalifa, H. S., & Al‐Salman, A. S. (2024) · IEEE Access · doi:10.1109/access.2024.3404668 · Cited by 5 · see citing works
GreekT5: Sequence-to-Sequence Models for Greek News Summarization
Giarelis, N., Mastrokostas, C., & Karacapilidis, N. I. (2024) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-63215-0_5 · Cited by 5 · see citing works
How Language-Dependent is Emotion Detection? Evidence from Multilingual BERT
Bruyne, L. D., Singh, P., Clercq, O. D., Lefever, E., & Hoste, V. (2022) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2022.mrl-1.7 · Cited by 5 · see citing works
Pérez, N., Cuadros, M., & Rigau, G. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102682 · Cited by 4 · see citing works
Almeida, T. S., Nogueira, R. F., & Pedrini, H. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5788 · Cited by 4 · see citing works
FinDeBERTaV2: Word-Segmentation-Free Pre-trained Language Model for Finance
Suzuki, M., Sakaji, H., Hirano, M., & Izumi, K. (2024) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.39-4_fin23-g · Cited by 4 · see citing works
- Authors
- Ortiz Suárez, P. J., Romary, L., & Sagot, B.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 1703-1714
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034617741
- Cited by
- 166
- References
- 0
- Citations by year
- 2026: 6 · 2025: 24 · 2024: 21 · 2023: 26 · 2022: 35 · 2021: 36 · 2020: 18
- FWCI
- 10.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche (ANR‐19, ANR-18-CE38-0003, ANR18, ANR-19-P3IA-0001)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- contextual word embeddings, ELMo, Common Crawl, Wikipedia, part-of-speech tagging, syntactic analysis, multilingual BERT, low-resource languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
