CoChat
Sign Up
Cited by 30Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Open access
Yong, Zheng Xin, Schoelkopf, Hailey, Muennighoff, Niklas, Aji, Alham Fikri, Adelani, David Ifeoluwa, Almubarak, Khalid, Bari, M Saiful, Sutawika, Lintang +7 more·2023·Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2023.acl-long.653
Open this paper in CoChat — free accountOpen PDF

Abstract

Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.acl-long.653

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most papers that came out of the BigScience Workshop used this functionality to report distributions of scores across different prompting set-ups.

    “Most papers that came out of the BigScience Workshop used this functionality to report distributions of scores across different prompting set-ups (Sanh et al., 2022; Muennighoff et al., 2022; Yong et al., 2023; BigScience Workshop et al., 2023).”

    From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Yong et al., 2023

    Section 3.5 and Appendix F describe zero-shot prompting with English and translated templates, and Appendix E discusses performance with an English-template setting. The Additional information section says the reported results are from single runs; it does not establish that this paper reported distributions of scores or that most BigScience Workshop papers did so. Page numbers are unavailable in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. 2020. In neural machine transla- tion, what does transfer learning transfer? In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7701-7710.
  2. Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1778-1796, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
  3. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. 2021. Efficient large scale language mod- eling with mixtures of experts. arXiv preprint arXiv:2112.10684. · arXiv
  4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics. · DOI
  5. Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguistics, 7:597- 610. · DOI
  6. Alexandre Berard. 2021. Continual learning in multilin- gual NMT via language-specific embeddings. In Proceedings of the Sixth Conference on Machine Translation, pages 542-565, Online. Association for Computational Linguistics.
  7. Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Puro- hit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open- source autoregressive language model. In Proceed- ings of BigScience Episode #5 -Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pages 95-136, virtual+Dublin. Association for Computational Linguistics. · DOI
  8. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324-1334, Online. Association for Computational Linguistics. · DOI · Paper page
  9. Luis Chiruzzo, Santiago Góngora, Aldo Alvarez, Gus- tavo Giménez-Lugo, Marvin Agüero-Torales, and Yliana Rodríguez. 2022. Jojajovai: A parallel Guarani-Spanish corpus for MT benchmarking. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2098-2107, Mar- seille, France. European Language Resources Asso- ciation.
  10. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics. · DOI
  11. Javier De la Rosa and Andrés Fernández. 2022. Zero- shot reading comprehension and reasoning for span- ish with bertin gpt-j-6b. Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022).
  12. Philipp Dufter and Hinrich Schütze. 2020. Identifying elements essential for BERT's multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4423-4437, Online. Association for Computa- tional Linguistics. · DOI
  13. Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 4555-4567, Online. Association for Computa- tional Linguistics. · DOI
  14. Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6279-6299, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
  15. Fahim Faisal and Antonios Anastasopoulos. 2022. Phylogeny-inspired adaptation of multilingual mod- els to new languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 434-452, Online only. Association for Computational Linguis- tics.
  16. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
  17. Jinlan Fu, See-Kiong Ng, and Pengfei Liu. 2022. Poly- glot prompt: Multilingual multitask promptraining. arXiv preprint arXiv:2204.14264. · arXiv
  18. Yoshinari Fujinuma, Jordan Boyd-Graber, and Katha- rina Kann. 2022. Match the script, adapt if multilin- gual: Analyzing the effect of multilingual pretraining on cross-lingual transferability. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1500-1512, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
  19. Philip Gage. 1994. A new algorithm for data compres- sion. C Users Journal, 12(2):23-38.
  20. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representa- tions.
  21. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
  22. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. arXiv preprint arXiv:2205.05638. · arXiv
  23. Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efficient language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 6253-6264, Dublin, Ireland. Associa- tion for Computational Linguistics. · DOI
  24. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 448-462, Online. Association for Computa- tional Linguistics. · DOI
  25. Martin Müller and Florian Laurent. 2022. Cedille: A large autoregressive french language model. arXiv preprint arXiv:2202.03371. · arXiv
  26. Graham Neubig and Junjie Hu. 2018. Rapid adapta- tion of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 875-880, Brussels, Belgium. Association for Com- putational Linguistics. · DOI
  27. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for process- ing huge corpora on medium to low resource infras- tructures. In Proceedings of the Workshop on Chal- lenges in the Management of Large Corpora (CMLC- 7) 2019. Cardiff, 22nd July 2019, pages 9 -16, Mannheim. Leibniz-Institut für Deutsche Sprache. · DOI
  28. Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Jun- seong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jung-Woo Ha, and Kyunghyun Cho. 2021. KLUE: Korean language understanding evalu- ation. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
  29. Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022. BAD-X: Bilingual adapters im- prove zero-shot cross-lingual transfer. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 1791- 1799, Seattle, United States. Association for Compu- tational Linguistics. · DOI
  30. Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics. · DOI
  31. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, Online. Association for Computa- tional Linguistics. · DOI
  32. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebas- tian Ruder. 2021b. UNKs everywhere: Adapting mul- tilingual language models to new scripts. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186-10203, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. · DOI
  33. Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. 2020. Monolingual adapters for zero-shot neural machine translation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4465-4470, Online. Association for Computational Linguistics. · DOI
  34. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online. As- sociation for Computational Linguistics. · DOI
  35. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Confer- ence on Learning Representations.
  36. Kunxun Qi, Hai Wan, Jianfeng Du, and Haolan Chen. 2022. Enhancing cross-lingual natural language in- ference by prompt-learning from cross-lingual tem- plates. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1910-1923, Dublin, Ire- land. Association for Computational Linguistics. · DOI
  37. Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  38. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725.
  39. Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilin- gual. · DOI
  40. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and mem- ory efficient transfer learning. arXiv preprint arXiv:2206.06522. · arXiv
  41. Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Train- ing neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems.
  42. Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luc- cioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the chal- lenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode #5 -Workshop on Challenges & Perspectives in Creating Large Lan- guage Models, pages 26-41, virtual+Dublin. Associ- ation for Computational Linguistics. · DOI
  43. Alexey Tikhonov and Max Ryabinin. 2021. It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534-3546, Online. Association for Computational Linguistics. · DOI
  44. Lifu Tu, Caiming Xiong, and Yingbo Zhou. 2022. Prompt-tuning can be much better than fine-tuning on cross-lingual understanding with multilingual lan- guage models. arXiv preprint arXiv:2210.12360. · arXiv
  45. Ahmet Üstün, Alexandre Berard, Laurent Besacier, and Matthias Gallé. 2021. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6650-6662, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. · DOI
  46. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low- resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649-2656, Online. Association for Computational Linguistics. · DOI · Paper page
  47. Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. 2022. Cross- lingual few-shot learning on unseen languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 777-791, Online only. Association for Computational Linguistics.
  48. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguis- tics. · DOI
  49. Zheng-Xin Yong and Vassilina Nikoulina. 2022. Adapt- ing bigscience multilingual model to unseen lan- guages. arXiv preprint arXiv:2204.04873. · arXiv
  50. Rong Zhang, Revanth Gangi Reddy, Md Arafat Sul- tan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020. Multi-stage pre-training for low- resource domain adaptation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5461-5468, Online. Association for Computational Linguistics. · DOI
  51. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. · arXiv
  52. Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547-8555, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. · DOI

Cited by

Sorted by citations

Authors
Yong, Z. X., Schoelkopf, H., Muennighoff, N., Aji, A. F., Adelani, D. I., Almubarak, K., Bari, M. S., Sutawika, L., Kasai, J., Baruwa, A., Winata, G., Biderman, S., Raff, E., Radev, D., & Nikoulina, V.
Affiliations
  • Brown University (US)
  • Yale University (US)
  • Hugging Face (US)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
  • University College London (GB)
  • Prince Sattam Bin Abdulaziz University (SA)
  • Nanyang Technological University (SG)
  • University of Washington (US)
  • University of Oregon (US)
  • Hong Kong University of Science and Technology (HK)
  • Booz Allen Hamilton (United States) (US)
ORCID
Venue
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2023
Pages
11682-11703
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2023.acl-long.653
OpenAlex
W4385570226
Cited by
30
References
10
Citations by year
2026: 2 · 2025: 6 · 2024: 11 · 2023: 11
FWCI
4.76
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • UCL Discovery (University College London) (submittedVersion, cc-by-nc-sa)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
bloom, multilingual language models, zero-shot prompting, language support, large language models, multilingual text generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review