CoChat
Sign Up
Cited by 176Natural Language Processing TechniquesTopic ModelingDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 5 of the 12 references it cites were found in a registry or an index, and 7 are not checked yet.

A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

Open access
Ortiz Suárez, Pedro Javier, Romary, Laurent, Sagot, Benoît·2020·Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
DOI 10.18653/v1/2020.acl-main.156
Open this paper in CoChat — free accountOpen PDF

Abstract

We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages.We then compare the performance of OSCARbased and Wikipedia-based ELMo embeddings for these languages on the part-ofspeech tagging and parsing tasks.We show that, despite the noise in the Common-Crawlbased OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia.They actually equal or improve the current state of the art in tagging and parsing for all five languages.In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the crosslingual benefit of multilingual embedding architectures.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.acl-main.156

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 24

  1. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1638-1649. Association for Computa- tional Linguistics.
  2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations 22 http://prairie-institute.fr/ 23 https://wiki.inria.fr/wikis/ ClustersSophia for multilingual NLP. In Proceedings of the Seven- teenth Conference on Computational Natural Lan- guage Learning, pages 183-192, Sofia, Bulgaria. Association for Computational Linguistics.
  3. Bernd Bohnet, Ryan McDonald, Gonçalo Simões, Daniel Andor, Emily Pitler, and Joshua Maynez. 2018. Morphosyntactic tagging with a meta- BiLSTM model over context sensitive token encod- ings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2642-2652, Melbourne, Australia. Association for Computational Linguis- tics.
  4. Matthias Buch-Kromann. 2003. The danish depen- dency treebank and the dtag treebank tool. In 2nd Workshop on Treebanks and Linguistic Theories (TLT), Sweden, pages 217-220.
  5. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55-64, Brussels, Belgium. Association for Compu- tational Linguistics.
  6. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
  7. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Ar- mand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the 11th Language Resources and Evaluation Con- ference, Miyazaki, Japan. European Language Re- source Association.
  8. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651. · arXiv
  9. Languages, 1 Model: Parsing Universal De- pendencies Universally. arXiv e-prints, page arXiv:1904.02099. · arXiv
  10. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv e- prints, page arXiv:1909.11942. · arXiv
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining ap- proach. CoRR, abs/1907.11692. · arXiv
  12. Louis Martin, Benjamin Muller, Pedro Javier Or- tiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. CamemBERT: a Tasty French Language Model. arXiv e-prints, page arXiv:1911.03894. · arXiv
  13. Rada Mihalcea. 2007. Using Wikipedia for automatic word sense disambiguation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computa- tional Linguistics; Proceedings of the Main Confer- ence, pages 196-203, Rochester, New York. Associ- ation for Computational Linguistics.
  14. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Ad- vances in pre-training distributed word representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018.
  15. Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2012. A universal part-of-speech tagset. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation, LREC 2012, Istan- bul, Turkey, May 23-25, 2012, pages 2089-2096. Eu- ropean Language Resources Association (ELRA).
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Explor- ing the Limits of Transfer Learning with a Uni- fied Text-to-Text Transformer. arXiv e-prints, page arXiv:1910.10683. · arXiv
  17. Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal dependency pars- ing with multi-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Pars- ing from Raw Text to Universal Dependencies, pages 113-123, Brussels, Belgium. Association for Com- putational Linguistics.
  18. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197-207, Brussels, Belgium. Association for Computational Linguistics. · DOI
  19. Milan Straka, Jana Straková, and Jan Hajic. 2019. Eval- uating contextualized embeddings on 54 languages in POS tagging, lemmatization and dependency pars- ing. CoRR, abs/1908.07448. · arXiv
  20. Mariona Taulé, Maria Antònia Martí, and Marta Re- casens. 2008. Ancora: Multilevel annotated corpora for catalan and spanish. In Proceedings of the In- ternational Conference on Language Resources and Evaluation, LREC 2008, 26 May -1 June 2008, Mar- rakech, Morocco. European Language Resources Association.
  21. Trieu H. Trinh and Quoc V. Le. 2018. A sim- ple method for commonsense reasoning. CoRR, abs/1806.02847. · arXiv
  22. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2019. CC- Net: Extracting High Quality Monolingual Datasets from Web Crawl Data. arXiv e-prints, page arXiv:1911.00359. · arXiv
  23. Fei Wu and Daniel S. Weld. 2010. Open information extraction using Wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Compu- tational Linguistics, pages 118-127, Uppsala, Swe- den. Association for Computational Linguistics.
  24. Daniel Zeman, Jan Hajič, Martin Popel, Martin Pot- thast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Mul- tilingual parsing from raw text to universal depen- dencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Univer- sal Dependencies, pages 1-21, Brussels, Belgium. Association for Computational Linguistics.

Cited by

Sorted by citations

Items 101–176 of 176

Related

See all

Metadata

Authors
Ortiz Suárez, P. J., Romary, L., & Sagot, B.
Affiliations
Institut national de recherche en sciences et technologies du numérique (FR)
ORCID
Venue
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Publisher
Association for Computational Linguistics
Published
2020
Pages
1703-1714
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2020.acl-main.156
OpenAlex
W3034617741
Cited by
166
References
0
Citations by year
2026: 6 · 2025: 24 · 2024: 21 · 2023: 26 · 2022: 35 · 2021: 36 · 2020: 18
FWCI
10.98
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
Agence Nationale de la Recherche (ANR‐19, ANR-18-CE38-0003, ANR18, ANR-19-P3IA-0001)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
contextual word embeddings, ELMo, Common Crawl, Wikipedia, part-of-speech tagging, syntactic analysis, multilingual BERT, low-resource languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review