The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
On the Cross-lingual Transferability of Monolingual Representations
Open accessAbstract
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot crosslingual setting.This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions.We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level.More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective-freezing parameters of all other layers.This approach does not rely on a shared vocabulary or joint training.However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD).Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages.We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Improving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
XNLI: Evaluating Cross-lingual Sentence Representations
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., & Stoyanov, V. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1269 · Cited by 915
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Learning multiple visual domains with residual adapters
Rebuffi, S.-A., Bilen, H., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1705.08045 · Cited by 605
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Wu, S., & Dredze, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1077 · Cited by 558
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Artetxe, M., Labaka, G., & Agirre, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1073 · Cited by 555
Self-citationLearning bilingual word embeddings with (almost) no bilingual data
Artetxe, M., Labaka, G., & Agirre, E. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1042 · Cited by 455
Self-citationWiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 406
Inducing Crosslingual Distributed Representations of Words
Klementiev, A., Titov, I., & Bhattarai, B. (2012) · International Conference on Computational Linguistics · Cited by 389
MLQA: Evaluating Cross-lingual Extractive Question Answering
Lewis, P., Oguz, B., Rinott, R., Riedel, S., & Schwenk, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.653 · Cited by 387
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334
Cross-Lingual Ability of Multilingual BERT: An Empirical StudyPreprint
Karthikeyan, K., Wang, Z., Mayhew, S., & Roth, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07840 · Cited by 329
PAWS: Paraphrase Adversaries from Word Scrambling
Zhang, Y., Baldridge, J., & He, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1131 · Cited by 308
Adversarial Training for Unsupervised Bilingual Lexicon Induction
Zhang, M., Liu, Y., Luan, H., & Sun, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1179 · Cited by 256
On the Limitations of Unsupervised Bilingual Dictionary Induction
Søgaard, A., Ruder, S., & Vulić, I. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1072 · Cited by 254
Self-citationHow Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks
Kaushik, D., & Lipton, Z. C. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1546 · Cited by 228
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Yang, Y., Zhang, Y., Tar, C., & Baldridge, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1382 · Cited by 223
Schuster, T., Ram, O., Barzilay, R., & Globerson, A. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1162 · Cited by 182
Glavaš, G., Litschko, R., Ruder, S., & Vulić, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1070 · Cited by 180
Self-citationMultilingual Alignment of Contextual Word RepresentationsPreprint
Cao, S., Kitaev, N., & Klein, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03518 · Cited by 156
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Eisenschlos, J. M., Ruder, S., Czapla, P., Kadras, M., Gugger, S., & Howard, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1572 · Cited by 95
Self-citationUnicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual TasksPreprint
Huang, H., Liang, Y., Duan, N., Gong, M., Shou, L., Jiang, D., & Zhou, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00964 · Cited by 94
From English To Foreign Languages: Transferring Pre-trained Language ModelsPreprint
Tran, K. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.07306 · Cited by 47
Emerging Cross-lingual Structure in Pretrained Language ModelsPreprint
Wu, S., Conneau, A., Li, H., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01464 · Cited by 21
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., & Hsieh, C.-J. (2019) · arXiv · doi:10.48550/arxiv.1904.00962
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Multilingual Denoising Pre-training for Neural Machine Translation
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., & Zettlemoyer, L. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00343 · Cited by 1,527
mT5: A massively multilingual pre-trained text-to-text transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 594
Metadata
- Authors
- Artetxe, M., Ruder, S., & Yogatama, D.
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 4623-4637
- OpenAlex
- W2982180741
- Cited by
- 598
- References
- 34
- FWCI
- 47.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.