CoChat
Sign Up
Cited by 191Topic ModelingEthics and Social Impacts of AINatural Language Processing Techniques

Having Beer after Prayer? Measuring Cultural Bias in Large Language Models

Open access
Naous, Tarek, Ryan, Michael J, Ritter, Alan, Xu, Wei·2024·Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2024.acl-long.862
Open this paper in CoChat — free accountOpen PDF

Abstract

As the reach of large language models (LMs) expands globally, their ability to cater to diverse cultural contexts becomes crucial.Despite advancements in multilingual capabilities, models are not designed with appropriate cultural nuances.In this paper, we show that multilingual and Arabic monolingual LMs exhibit bias towards entities associated with Western culture.We introduce CAMeL, a novel resource of 628 naturally-occurring prompts and 20,368 entities spanning eight types that contrast Arab and Western cultures.CAMeL provides a foundation for measuring cultural biases in LMs through both extrinsic and intrinsic evaluations.Using CAMeL, we examine the cross-cultural performance in Arabic of 16 different LMs on tasks such as story generation, NER, and sentiment analysis, where we find concerning cases of stereotyping and cultural unfairness.We further test their text-infilling performance, revealing the incapability of appropriate adaptation to Arab cultural contexts.Finally, we analyze 6 Arabic pre-training corpora and find that commonly used sources such as Wikipedia may not be best suited to build culturally aware LMs, if used as they are without adjustment.We will make CAMeL publicly available at: https://github.com/tareknaous/camel

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.acl-long.862

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models are frequently trained on datasets predominantly sourced from WEIRD contexts.

    “Large language models are frequently trained on datasets predominantly sourced from WEIRD contexts [83].”

    From Generative AI Mental Health Chatbots as Therapeutic Tools: Systematic Review and Meta-Analysis of Their Role in Reducing Mental Health Issues (2025) · cites this paper as [83]

    In “Analyzing Arabic Pre-training Data” and the following results discussion (page unavailable), the paper examines six Arabic corpora commonly used for language-model pre-training and finds that Wikipedia, international news, and web-crawled data are relatively Western-centric. It does not establish that large language models generally are trained on datasets predominantly sourced from WEIRD contexts; Western content in some Arabic corpora is narrower than that claim. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 94

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demon- strations, pages 11-16.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, et al. 2021. ARBERT & MARBERT: Deep bidi- rectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088-7105.
  3. Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, and Natasha Jaques. 2023. Moral foundations of large language models. arXiv preprint arXiv:2310.15337. · arXiv
  4. Abubakar Abid, Maheen Farooqi, and James Zou. 2021a. Large language models associate muslims with vio- lence. Nature Machine Intelligence, 3(6):461-463.
  5. Abubakar Abid, Maheen Farooqi, and James Zou. 2021b. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES '21, page 298-306, New York, NY, USA. Association for Computing Machinery.
  6. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528. · arXiv
  7. Haozhe An, Zongxia Li, Jieyu Zhao, and Rachel Rudinger. 2023. SODAPOP: Open-ended discov- ery of social biases in social commonsense reasoning models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 1565-1588.
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic lan- guage understanding. In LREC Workshop Language Resources and Evaluation Conference 11-16 May 2020, page 9.
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraGPT2: Pre-trained transformer for arabic lan- guage generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196- 207.
  10. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augen- stein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114-130.
  11. Parishad BehnamGhader and Aristides Milios. 2022. An analysis of social biases present in BERT variants across multiple languages. In Workshop on Trust- worthy and Socially Responsible Machine Learning, NeurIPS 2022.
  12. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named en- tity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143-153. Springer.
  13. Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022. Re- contextualizing fairness in nlp: The case of india. In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 727-740.
  14. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  15. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from lan- guage corpora contain human-like biases. Science, 356(6334):183-186.
  16. Yang Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger, and Linda Zou. 2022a. Theory-grounded measurement of us social stereotypes in english lan- guage models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1276-1295.
  17. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and hu- man societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53-67.
  18. Laura Chiticariu, Yunyao Li, and Frederick R. Reiss. 2013. Rule-based information extraction is dead! long live rule-based information extraction systems! In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 827-832, Seattle, Washington, USA. Association for Computational Linguistics.
  19. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 8440-8451.
  20. Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. 2021. Quantifying social biases in nlp: A generaliza- tion and empirical comparison of extrinsic fairness metrics. Transactions of the Association for Compu- tational Linguistics, 9:1249-1267.
  21. Dipto Das, Shion Guha, and Bryan Semaan. 2023. To- ward cultural bias evaluation datasets: The case of bengali gender, religious, and national identity. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 68-83.
  22. Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Sriku- mar. 2021. OSCaR: Orthogonal subspace correc- tion and rectification of biases in word embeddings. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5034-5050.
  23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
  24. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the repre- sentation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388. · arXiv
  25. Ibrahim Abu El-Khair. 2016. 1.5 billion words Arabic corpus. arXiv preprint arXiv:1611.04033. · arXiv
  26. AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A chal- lenging benchmark for Arabic language understand- ing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559-9586, Toronto, Canada. Association for Computational Linguistics. · DOI
  27. Ashraf Elnagar, Yasmin S Khalifa, and Anas Einea. 2018. Hotel arabic-reviews dataset construction for sentiment analysis applications. Intelligent natural language processing: Trends and applications, pages 35-52.
  28. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv preprint arXiv:2305.08283. · arXiv
  29. Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi's moral philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 26-42.
  30. Dayne Freitag, John Cadigan, Robert Sasseen, and Paul Kalmar. 2022. Valet: Rule-based information ex- traction for rapid deployment. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 524-533, Marseille, France. European Language Resources Association.
  31. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Der- noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770. · arXiv
  32. Adriano Gómez-Bantel. 2018. Football clubs as symbols of regional identities. In Football, Community and Sustainability, pages 32-42. Routledge.
  33. Jesse Graham, Brian A Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H Ditto. 2011. Mapping the moral domain. Journal of personality and social psychology, 101(2):366.
  34. Valeschka M Guerra and Roger Giner-Sorolla. 2010. The community, autonomy, and divinity scale (cads): A new tool for the cross-cultural study of morality. Journal of cross-cultural psychology, 41(1):35-50.
  35. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embed- dings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122-133.
  36. Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin A Rothkopf, Alexander Fraser, and Kristian Kerst- ing. 2022. Speaking multiple languages affects the moral bias of language models. arXiv preprint arXiv:2211.07733. · arXiv
  37. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversa- tional ai: Converging evidence on chatgpt's pro- environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768. · arXiv
  38. Babak Hemmatian, Razan Baltaji, and Lav R Varshney. 2023. Muslim-violence bias persists in debiased gpt models. arXiv preprint arXiv:2310.18368. · arXiv
  39. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. 2022. Challenges and strategies in cross-cultural nlp. In 60th Annual Meet- ing of the Association-for-Computational-Linguistics (ACL), MAY 22-27, 2022, Dublin, IRELAND, pages 6997-7013. Association for Computational Linguis- tics.
  40. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  41. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023. AceGPT, localizing large language models in Arabic. arXiv preprint arXiv:2309.12053. · arXiv
  42. Jing Huang and Diyi Yang. 2023. Culturally aware natu- ral language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591-7609.
  43. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92- 104.
  44. Rebecca L Johnson, Giada Pistilli, Natalia Menédez- González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The Ghost in the Machine has an American ac- cent: value conflict in GPT-3. arXiv preprint arXiv:2203.07785. · arXiv
  45. Masahiro Kaneko and Danushka Bollegala. 2022. Un- masking the mask-evaluating social biases in masked language models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11954-11962.
  46. Amr Keleg and Walid Magdy. 2023. DLAMA: A frame- work for curating culturally diverse facts for probing the knowledge of pretrained language models. arXiv preprint arXiv:2306.05076. · arXiv
  47. Alex Koch, Roland Imhoff, Ron Dotsch, Christian Un- kelbach, and Hans Alves. 2016. The abc of stereo- types about groups: Agency/socioeconomic suc- cess, conservative-progressive beliefs, and commu- nion. Journal of personality and social psychology, 110(5):675.
  48. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727-4734.
  49. Anne Lauscher and Goran Glavaš. 2019. Are we con- sistently biased? multidimensional analysis of biases in distributional word vectors. In Proceedings of the Eighth Joint Conference on Lexical and Computa- tional Semantics, pages 85-91, Minneapolis, Min- nesota. Association for Computational Linguistics.
  50. Nayeon Lee, Chani Jung, and Alice Oh. 2023. Hate speech classifiers are culturally insensitive. In Pro- ceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 35-46.
  51. Sharon Levy, Neha Anna John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vit- torio Castelli, and Dan Roth. 2023. Comparing biases and the impact of multilingual training across multiple languages. arXiv preprint arXiv:2305.11242. · arXiv
  52. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023. A survey on fairness in large language models. arXiv preprint arXiv:2308.10149. · arXiv
  53. Weicheng Ma, Brian Chiang, Tong Wu, Lili Wang, and Soroush Vosoughi. 2023a. Intersectional stereotypes in large language models: Dataset and analysis. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 8589-8597.
  54. Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, An- drew Koulogeorge, Lili Wang, Diyi Yang, and Soroush Vosoughi. 2023b. Deciphering stereotypes in pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11328-11345.
  55. Marta Marchiori Manerba, Karolina Stańczak, Riccardo Guidotti, and Isabelle Augenstein. 2023. Social bias probing: Fairness benchmarking for language models. arXiv preprint arXiv:2311.09090. · arXiv
  56. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explana- tory analysis based on hofstede's cultural dimensions. arXiv preprint arXiv:2309.12342. · arXiv
  57. Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628.
  58. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064.
  59. Massimo Montanari. 2006. Food is culture. Columbia University Press.
  60. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021a. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356-5371.
  61. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020b. Crows-pairs: A chal- lenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133. · arXiv
  62. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907-1917.
  63. Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952-9962.
  64. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
  65. Aida Ramezani and Yang Xu. 2023. Knowledge of cultural moral norms in large language models. arXiv preprint arXiv:2306.01857. · arXiv
  66. Brian Roark, Richard Sproat, Cyril Allauzen, Michael Riley, Jeffrey Sorensen, and Terry Tai. 2012. The opengrm open-source finite-state grammar software libraries. In Proceedings of the ACL 2012 System Demonstrations, pages 61-66.
  67. Candace Ross, Boris Katz, and Andrei Barbu. 2021. Measuring social biases in grounded vision and lan- guage embeddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 998-1008.
  68. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712.
  69. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. · arXiv
  70. Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. · arXiv
  71. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human- like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258-268.
  72. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. · arXiv
  73. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4275-4293.
  74. Vered Shwartz. 2022. Good night at 4 pm?! time expressions in different cultures. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2842-2853.
  75. Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing. arXiv preprint arXiv:2112.14168. · arXiv
  76. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastruc- tures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz- Institut für Deutsche Sprache.
  77. Magdalena Szumilas. 2010. Explaining odds ratios. Journal of the Canadian academy of child and ado- lescent psychiatry, 19(3):227.
  78. Fatjri Nur Tajuddin. 2018. Cultural and social identity in clothing matters "different cultures, different mean- ings". European Journal of Behavioral Sciences, 1(4):21-25.
  79. Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in neural information processing systems, 32.
  80. Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A shocking amount of the web is machine translated: Insights from multi-way parallelism. arXiv preprint arXiv:2401.05749. · arXiv
  81. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  82. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei- Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827. · arXiv
  83. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
  84. Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023. On evaluating and mitigating gender biases in multilingual settings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 307-318.
  85. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Pan- chanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Nationality bias in text generation. In Proceed- ings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 116-122.
  86. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
  87. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. "kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3730-3748.
  88. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023a. GPT-NER: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428. · arXiv
  89. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R Lyu. 2023b. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. arXiv preprint arXiv:2310.12481. · arXiv
  90. Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, and Pengfei Liu. 2023. Align on the fly: Adapting chatbot behavior to established norms. arXiv preprint arXiv:2312.15907. · arXiv
  91. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
  92. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai-Wei Chang. 2022. GeoM- LAMA: Geo-diverse commonsense probing on multi- lingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039-2055.
  93. Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muen- nighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. 2022. Bloom+1: Adding language support to bloom for zero-shot prompting. arXiv preprint arXiv:2212.09535. · arXiv
  94. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus-preparation and integration into the CLARIN-infrastructure. In Pro- ceedings of the fourth arabic natural language pro- cessing workshop, pages 175-182.

Cited by

Sorted by citations

Related

Authors
Naous, T., Ryan, M. J., Ritter, A., & Xu, W.
Affiliations
Georgia Institute of Technology (US)
ORCID
Venue
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2024
Pages
16366-16393
ISSN
0736-587X (ISSN-L)
DOI
10.18653/v1/2024.acl-long.862
OpenAlex
W4402671169
Cited by
96
References
0
Citations by year
2026: 40 · 2025: 42 · 2024: 10 · 2023: 4
FWCI
27.15
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2052498, 2144493)
  • Intelligence Advanced Research Projects Activity
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
Arabic language models, cross-cultural assessment, text generation, named entity recognition, sentiment analysis, text infilling, cultural fairness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review