CoChat
Sign Up
Cited by 44Natural Language Processing TechniquesTopic ModelingSyntax, Semantics, Linguistic Variation

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank

Open access
Chau, Ethan C., Lin, Lucy H., Smith, Noah A.·2020·Findings of the Association for Computational Linguistics: EMNLP 2020
DOI 10.18653/v1/2020.findings-emnlp.118
Open this paper in CoChat — free accountOpen PDF

Abstract

Pretrained multilingual contextual representations have shown great success, but due to the limits of their pretraining data, their benefits do not apply equally to all language varieties.This presents a challenge for language varieties unfamiliar to these models, whose labeled and unlabeled data is too limited to train a monolingual model effectively.We propose the use of additional language-specific pretraining and vocabulary augmentation to adapt multilingual models to low-resource settings.Using dependency parsing of four diverse low-resource language varieties as a case study, we show that these methods significantly improve performance over baselines, especially in the lowestresource cases, and demonstrate the importance of the relationship between such models' pretraining data and target language varieties.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.findings-emnlp.118

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Vocabulary expansion adds tokens to the existing vocabulary and increases the embedding matrix size.

    “Expansion adds tokens to the existing vocabulary, increasing embedding matrix size: Chau et al. (2020) extend multilingual BERT; Kim et al. (2024) add 8,960 Korean tokens to SOLAR-10.7B with progressive unfreezing; Wang et al. (2020) extend to low-resource languages.”

    From Writing-System-Level Tokenizer Adaptation for Byte-Level BPE (2026) · cites this paper as Chau et al., 2020

    The Vocabulary Augmentation section says Chau et al. add language-specific wordpieces, but its footnote says they repurpose 99 unused slots in mBERT’s fixed-size vocabulary without modifying the pretrained model. The supplementary material likewise says the new wordpieces fill those slots. Thus, this paper does not support the attributed increase in embedding-matrix size. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for Basque. In Proc. of LREC.
  2. Emily Alsentzer, John Murphy, William Boag, Wei- Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clini- cal BERT embeddings. In Proc. of Clinical Natural Language Processing Workshop. · DOI · Paper page
  3. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proc. of ACL. · DOI
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146. · DOI · Paper page
  5. Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. In Proc. of ICLR.
  6. Tuhin Chakrabarty, Christopher Hidey, and Kathy McK- eown. 2019. IMHO fine-tuning improves claim de- tection. In Proc. of NAACL-HLT. · DOI
  7. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proc. of ACL. · DOI
  8. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proc. of EMNLP-IJCNLP. · DOI · Paper page
  9. Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In Proc. of ICLR.
  10. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language pro- cessing platform. In Proc. of Workshop for NLP Open Source Software (NLP-OSS). · DOI
  11. Xiaochuang Han and Jacob Eisenstein. 2019. Unsuper- vised domain adaptation of contextualized embed- dings for sequence labeling. In Proc. of EMNLP. · DOI
  12. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proc. of ACL. · DOI · Paper page
  13. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In Proc. of ICLR.
  14. Dan Kondratyuk and Milan Straka. 2019. 75 languages, 1 model: Parsing Universal Dependencies universally. In Proc. of EMNLP-IJCNLP. · DOI
  15. Surafel M. Lakew, Aliia Erofeeva, Matteo Negri, Mar- cello Federico, and Marco Turchi. 2018. Transfer learning in multilingual neural machine translation with dynamic vocabulary. In Proc. of IWSLT.
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692 [cs.CL]. · arXiv
  17. Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proc. of ACL. · DOI
  18. David Mueller, Nicholas Andrews, and Mark Dredze. 2020. Sources of transfer in multilingual named entity recognition. In Proc. of ACL. · DOI
  19. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019a. Low-resource parsing with crosslingual con- textualized representations. In Proc. of CoNLL. · DOI
  20. Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019b. Polyglot contextual representations improve crosslingual transfer. In Proc. of NAACL-HLT. · DOI
  21. Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019. Is multilingual BERT fluent in language generation? In Proc. of the First NLPL Workshop on Deep Learning for Natural Language Processing.
  22. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contex- tual word embeddings, with applications to zero-shot dependency parsing. In Proc. of NAACL-HLT. · DOI
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL. · DOI · Paper page
  24. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification? arXiv:1905.05583 [cs.CL]. · arXiv
  25. Peter Trudgill. 2003. A Glossary of Sociolinguistics. Edinburgh University Press. · DOI
  26. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Ari- vazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proc. of EMNLP-IJCNLP. · DOI
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS.
  28. Hongmin Wang, Jie Yang, and Yue Zhang. 2019. From genesis to creole language: Transfer learning for singlish universal dependencies parsing and pos tag- ging. ACM Transactions on Asian and Low-Resource Language Information Processing, 19(1). · DOI
  29. Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chieu. 2017. Universal dependencies parsing for colloquial Singaporean En- glish. In Proc. of ACL. · DOI
  30. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proc. of EMNLP-IJCNLP. · DOI
  31. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proc. of the 5th Workshop on Representation Learning for NLP. · DOI
  32. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv:1609.08144 [cs.CL]. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Chau, E. C., Lin, L. H., & Smith, N. A.
Affiliations
University of Washington (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2020
Publisher
Association for Computational Linguistics
Published
2020
Pages
1324-1334
DOI
10.18653/v1/2020.findings-emnlp.118
OpenAlex
W3089959042
Cited by
44
References
7
Citations by year
2026: 1 · 2025: 2 · 2024: 3 · 2023: 9 · 2022: 19 · 2021: 9 · 2020: 1
FWCI
3.62
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1813153)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
Keywords
multilingual BERT, dependency parsing, low-resource languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review