CoChat
Sign Up
PreprintCited by 6Topic ModelingSentiment Analysis and Opinion MiningLanguage and Culture

Multilingual Language Models are not Multicultural: A Case Study in Emotion

Open access
Havaldar, Shreya, Rai, Sunny, Singhal, Bhumika, Liu, Langchen, Guntuku, Sharath Chandra, Ungar, Lyle·2023·arXiv
DOI 10.48550/arxiv.2307.01370arXiv 2307.01370
Open this paper in CoChat — free accountOpen PDF

Abstract

Emotions are experienced and expressed differently across the world. In order to use Large Language Models (LMs) for multilingual tasks that require emotional sensitivity, LMs must reflect this cultural variation in emotion. In this study, we investigate whether the widely-used multilingual LMs in 2023 reflect differences in emotional expressions across cultures and languages. We find that embeddings obtained from LMs (e.g., XLM-RoBERTa) are Anglocentric, and generative LMs (e.g., ChatGPT) reflect Western norms, even when responding to prompts in other languages. Our results show that multilingual LMs do not successfully learn the culturally appropriate nuances of emotion and we highlight possible research directions towards correcting this.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.01370

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The paper says current models exhibit deficits in emotional reasoning, including emotional intelligence, affective bias, and understanding cultural variation in emotional expression.

    “current models exhibit deficits in emotional reasoning. This includes difficulties in emotional intelligence (EI) (Sabour et al., 2024; Hu et al., 2025; Amirizaniani et al., 2024b; Vzorinab et al., 2024), susceptibility to affective bias (Chochlakis et al., 2024), and limited understanding of cultural variations in emotional expression and interpretation (Havaldar et al., 2023).”

    From Large Language Model Reasoning Failures (2026) · cites this paper as Havaldar et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 6

Research works printed but not linked: 27

  1. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augen- stein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considera- tions in NLP (C3NLP), pages 114-130, Dubrovnik, Croatia. Association for Computational Linguistics.
  2. Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. XLM-EMO: Multilingual emotion prediction in so- cial media text. In Proceedings of the 12th Work- shop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 195-203, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
  3. Sven Buechel, Susanna Rücker, and Udo Hahn. 2020. Learning and evaluating emotion lexicons for 91 lan- guages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 1202-1217, Online. Association for Computa- tional Linguistics. · DOI
  4. Tianyu Cho and Kei Sawada. 2021. Pre-learning model for japanese natural language processing. Japanese Society for Artificial Intelligence Research Group Material Language/Speech Understanding and Dia- logue Processing, 93:169-170. · DOI
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. · arXiv
  6. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668, Online. Association for Computa- tional Linguistics.
  7. Javier De la Rosa, Eduardo G. Ponferrada, Manu Romero, Paulo Villegas, Pablo González de Prado Salas, and María Grandury. 2022. Bertin: Efficient pre-training of a spanish language model using perplexity sampling. Procesamiento del Lenguaje Natural, 68(0):13-23.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805. · arXiv
  9. Paul Ekman et al. 1999. Basic emotions. Handbook of cognition and emotion, 98(45-60):16.
  10. Emi Furukawa, June Tangney, and Fumiko Higashibara. 2012. Cross-cultural continuities and discontinuities in shame, guilt, and pride: A study of children re- siding in japan, korea and the usa. Self and Identity, 11(1):90-113.
  11. Fred Genesee. 1982. The social psychological signifi- cance of code switching in cross-cultural communi- cation. Journal of language and social psychology, 1(1):1-27.
  12. Elizabeth A Kensinger and Daniel L Schacter. 2006. Processing emotional pictures and words: Effects of valence and arousal. Cognitive, Affective, & Behav- ioral Neuroscience, 6(2):110-126.
  13. Shinobu Kitayama, Hazel Rose Markus, and Masaru Kurokawa. 2000. Culture, emotion, and well-being: Good feelings in japan and the united states. Cogni- tion & Emotion, 14(1):93-124.
  14. Michael Lewis, Kiyoko Takai-Kawakami, Kiyobumi Kawakami, and Margaret Wolan Sullivan. 2010. Cul- tural differences in emotional responses to success and failure. International journal of behavioral de- velopment, 34(1):53-61.
  15. Nangyeon Lim. 2016. Cultural differences in emotion: differences in emotional arousal level between the east and the west. Integrative Medicine Research, 5(2):105-109. · DOI
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. · arXiv
  17. Gabriel Magno and Virgilio Almeida. 2021. Measuring international online human values with word embed- dings. ACM Trans. Web, 16(2). · DOI
  18. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
  19. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  20. James A Russell. 1980. A circumplex model of af- fect. Journal of personality and social psychology, 39(6):1161.
  21. James A Russell, Maria Lewicka, and Toomas Niit. 1989. A cross-cultural study of a circumplex model of affect. Journal of personality and social psychol- ogy, 57(5):848.
  22. Hanieh Shakeri, Carman Neustaedter, and Steve Di- Paola. 2021. Saga: Collaborative storytelling with gpt-3. In Companion Publication of the 2021 Con- ference on Computer Supported Cooperative Work and Social Computing, CSCW '21, page 163-166, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
  23. Rene M Stulz and Rohan Williamson. 2003. Culture, openness, and finance. Journal of financial Eco- nomics, 70(3):313-349.
  24. Jonas Thiergart, Stefan Huber, and Thomas Übellacker. 2021. Understanding emails and drafting responses - an approach using GPT-3. CoRR, abs/2102.03062. · arXiv
  25. Jeanne L Tsai, Robert W Levenson, and Kimberly Mc- Coy. 2006. Cultural and temperamental variation in emotional response. Emotion, 6(3):484.
  26. Lu Wang, Munif Ishad Mujib, Jake Ryland Williams, George Demiris, and Jina Huh-Yoo. 2021. An evalu- ation of generative pre-training model-based therapy chatbot for caregivers. CoRR, abs/2107.13115. · arXiv
  27. Anna Wierzbicka. 1999. Emotions across languages and cultures: Diversity and universals. Cambridge university press.

Authors
Havaldar, S., Rai, S., Singhal, B., Liu, L., Guntuku, S. C., & Ungar, L.
Affiliations
University of Pennsylvania (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.01370
arXiv
2307.01370
OpenAlex
W4383468829
Cited by
6
References
0
Citations by year
2026: 1 · 2025: 2 · 2024: 1 · 2023: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Linguistics and Language › Language and Culture
Keywords
multilingual language models, cultural variation in emotion, emotional expression, Anglocentrism, XLM-R, ChatGPT, emotion embeddings, cross-cultural bias, emotional sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review