Meta-Learning for Low-Resource Neural Machine Translation
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML, Finn et al., 2017) for lowresource neural machine translation (NMT).We frame low-resource translation as a metalearning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks.We use the universal lexical representation (Gu et al., 2018b) to overcome the input-output mismatch across different languages.We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr and Ko) as target tasks.We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach (Zoph et al., 2016) and enables us to train a competitive NMT system with only a fraction of training examples.For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT'16 by seeing only 16,000 translated words (⇠ 600 parallel sentences).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Self-citationEnriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 1×
“We then estimate word vectors using fast-Text (Bojanowski et al., 2016) and align them across all the languages in an unsupervised way using MUSE (Conneau et al., 2018) to get multilingual word vectors.”
MethodA Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 4×
“Following Finn et al. (2017), we achieve this goal by defining the meta-objective function as where k ∼ U ( { 1 , . . . , K } ) refers to one meta-learning episode, and D T , D follow the uniform distribution over T ’s data.”
BackgroundMethodInfluentialOn the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Self-citationPrototypical Networks for Few-shot Learning
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 4,755
Cited by
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Meta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
“Many more tasks have since been tackled, including text classification [135], neural program induction [263] and synthesis [264], English to SQL program synthesis [265], text-based relationship graph extractor [266], machine translation [267], and quickly adapting to new personas in dialogue”
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
Wu, C.-S., Madotto, A., Hosseini-Asl, E., Xiong, C., Socher, R., & Fung, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1078 · Cited by 415
Advances and Challenges in Meta-Learning: A Technical Review
Vettoruzzo, A., Bouguelia, M.-R., Vanschoren, J., Rögnvaldsson, T., & Santosh, K. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357847 · Cited by 314
Neural Machine Translation for Low-resource Languages: A Survey
Ranathunga, S., Lee, E.-S. A., Prifti Skenduli, M., Shekhar, R., Alam, M., & Kaur, R. (2023) · ACM Computing Surveys · doi:10.1145/3567592 · Cited by 313
A Survey of Multilingual Neural Machine Translation
Dabre, R., Chu, C., & Kunchukuttan, A. (2021) · ACM Computing Surveys · doi:10.1145/3406095 · Cited by 294
Progress in Neural NLP: Modeling, Learning, and Reasoning
Zhou, M., Duan, N., Liu, S., & Shum, H.-Y. (2020) · Engineering · doi:10.1016/j.eng.2019.12.014 · Cited by 281
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279
“This is also reflected in other works on low-resource machine translation where Any → En typically bene-fits the most (Zoph et al., 2016; Nguyen and Chi-ang, 2017; Gu et al., 2018a,b).”
BackgroundDeep reinforcement learning: a survey
Wang, H.-N., Liu, N., Zhang, Y.-Y., Feng, D.-W., Huang, F., Li, D.-S., & Zhang, Y.-M. (2020) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1900533 · Cited by 274
Personalizing Dialogue Agents via Meta-Learning
Madotto, A., Lin, Z., Wu, C.-S., & Fung, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1542 · Cited by 183
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks
Zhou, D.-W., Ye, H.-J., Ma, L., Xie, D., Pu, S., & Zhan, D.-C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3200865 · Cited by 162
“MAML is widely applied into vast learning scenarios, e.g., robotics con-trol [49], neural translation [50] and uncertainty estimation [51], and many improved algorithms [52], [53], [54], [55], [56] have been made to further boost its performance.”
BackgroundFew-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107
“…– for example, GEN P ET requires manually designed patterns and some amount of unlabeled examples, whereas meta learning approaches (e.g., Gu et al., 2018; Dou et al., 2019; Qian and Yu, 2019) require large annotated datasets for related tasks –, or they cannot be transferred to a…”
BackgroundUniversal linguistic inductive biases via meta-learning
McCoy, R. T., Grant, E., Smolensky, P., Griffiths, T. L., & Linzen, T. (2020) · arXiv · doi:10.48550/arxiv.2006.16324 · Cited by 13
“…meta-learning from a cognitive perspective (Lake et al., 2017; Griffiths et al., 2019; Lake, 2019; Grant et al., 2019), and in applied settings meta-learning has been applied to language to create technology in low-resource languages (Gu et al., 2018; Ponti et al., 2019; Kann et al., 2020).”
BackgroundCrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
Metadata
- Authors
- Gu, J., Wang, Y., Chen, Y., Li, V. O. K., & Cho, K.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 3622-3631
- OpenAlex
- W2888541716
- Cited by
- 325
- References
- 39
- FWCI
- 20.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.