CoChat
Sign Up
PreprintCited by 1Ethics and Social Impacts of AILaw, AI, and Intellectual PropertyExplainable Artificial Intelligence (XAI)

Back to the Communities: A Mixed-Methods and Community-Driven Evaluation of Cultural Sensitivity in Text-to-Image Models

Open access
Kiden, Sarah, Peter, Oriane, Reyes-Cruz, Gisela, Klyshbekova, Maira, Choi, Sena, Bergin, Aislinn Gomez, Waheed, Maria, Eke, Damian +6 more·2025·arXiv
DOI 10.48550/arxiv.2510.27361arXiv 2510.27361
Open this paper in CoChat — free accountOpen PDF

Abstract

Evidence shows that text-to-image (T2I) models disproportionately reflect Western cultural norms, amplifying misrepresentation and harms to minority groups. However, evaluating cultural sensitivity is inherently complex due to its fluid and multifaceted nature. This paper draws on a state-of-the-art review and co-creation workshops involving 59 individuals from 19 different countries. We developed and validated a mixed-methods community-based evaluation methodology to assess cultural sensitivity in T2I models, which embraces first-person methods. Quantitative scores and qualitative inquiries expose convergence and disagreement within and across communities, illuminate the downstream consequences of misrepresentation, and trace how training data shaped by unequal power relations distort depictions. Extensive assessments are constrained by high resource requirements and the dynamic nature of culture, a tension we alleviate through a context-based and iterative methodology. The paper provides actionable recommendations for stakeholders, highlighting pathways to investigate the sources, mechanisms, and impacts of cultural (mis)representation in T2I models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.27361

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works argue against treatments of representation as a static concept that can be evaluated objectively.

    “This echoes recent works [10, 68, 89, 90] that argue against treatments of representation as a static concept that can be evaluated objectively.”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [68]

  • In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation or allow community members to self-select their participation, albeit with some limitations.

    “In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation (e.g., across geographical regions [52, 68, 91] and through local research partners [89]), or allow community members to selfselect their participation, albeit with some limitations (e.g., adults, higher-education levels or topic expertise, English-language ability”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [68]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 78

  1. [1][1] Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards Measuring and Modeling 'Culture' in LLMs: A Survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 15763-15784. doi:10.18653/v1/2024.emnlp-main.882 - · DOI
  2. [2][2] Bang An, Mucong Ding, Tahseen Rabbani, Aakriti Agrawal, Yuancheng Xu, Chenghao Deng, Sicheng Zhu, Abdirisak Mohamed, Yuxin Wen, Tom Goldstein, and Furong Huang. 2024. WAVES: Benchmarking the Robustness of Image Watermarks. arXiv preprint arXiv:2401.08573 (2024). doi:10.48550/arXiv.2401.08573 - · DOI
  3. [3][3] Mohammad Atari, Mona J. Xue, Peter S. Park, Damián Blasi, and Joseph Henrich. 2023. Which Humans? psyarxiv:10.31234 doi:10.31234/osf.io/5b26t - · DOI · Paper page
  4. [4][4] Anton Bakhtin, Arthur Szlam, and Marc'Aurelio Ranzato. 2018. GenEval: A Benchmark Suite for Evaluating Generative Models. (Sept. 2018). https://openreview.net/forum?id=HJeRm3Aqt7 -
  5. [5][5] Erin S. Barry, Jerusalem Merkebu, and Lara Varpio. 2022. Understanding State-of-the-Art Literature Reviews. J Grad Med Educ 14, 6 (Dec. 2022), 659-662. doi:10.4300/JGME-D-22-00705.1 - · DOI
  6. [6][6] Abhipsa Basu, R Venkatesh Babu, and Danish Pruthi. 2023. Inspecting the geographical representativeness of images from text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 5136-5147. -
  7. [7][7] Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of 'Bias' in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 5454-5476. doi:10.18653/v1/2020.acl-main.485 - · DOI · Paper page
  8. [8][8] Rishi Bommasani, Kathleen A. Creel, Ananya Kumar, Dan Jurafsky, and Percy Liang. 2022. Picking on the same person: does algorithmic monoculture lead to outcome homogenization?. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS '22) . Curran Associates Inc., Red Hook, NY, USA, 3663-3678. -
  9. [9][9] Florence Bonacina-Pugh, Ildegrada da Costa Cabral, and Jing Huang. 2021. Translanguaging in Education. Language Teaching 54, 4 (Oct. 2021), 439-471. doi:10.1017/S0261444821000173 - · DOI · Paper page
  10. [10][10] Virginia Braun and Victoria Clarke. 2021. Thematic Analysis: A Practical Guide (1st ed.). SAGE Publications Ltd, 1 Oliver's Yard, 55 City Road, London EC1Y 1SP. -
  11. [11][11] Tanujit Chakraborty, Ujjwal Reddy K. S, Shraddha M. Naik, Madhurima Panja, and Bayapureddy Manvitha. 2024. Ten Years of Generative Adversarial Nets (GANs): A Survey of the State-of-the-Art. Mach. Learn.: Sci. Technol. 5, 1 (March 2024), 011001. arXiv:2308.16316 [cs] doi:10.1088/26322153/ad1f77 - · DOI
  12. [12][12] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: an open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML'24, Vol. 235) . JMLR.org, Vienna, Austria, 8359-8388. -
  13. [14][14] Aditya Chinchure, Pushkar Shukla, Gaurav Bhatt, Kiri Salij, Kartik Hosanagar, Leonid Sigal, and Matthew Turk. 2024. TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models. In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXIX . Springer-Verlag, Berlin, Heidelberg, 429-446. doi:10.1007/978-3-031-72986-7\_25 - · DOI
  14. [15][15] Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . 3020-3031. doi:10.1109/ICCV51070.2023.00283 - · DOI
  15. [17][17] John W. Creswell and Cheryl N. Poth. 2017. Qualitative Inquiry and Research Design (fourth ed.). SAGE Publications Inc. -
  16. [18][18] Robert Curedale. 2013. Interviews, Observation and Focus Groups: 110 Methods for User-centered Design (first ed.). Design Community College Inc, Topanga, CA, USA. -
  17. [19][19] Fabio de Almeida and Sónia Rafael. 2024. Bias by Default.: Neocolonial Visual Vocabularies in AI Image Generating Design Practices.. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA '24) . Association for Computing Machinery, New York, NY, USA, 1-8. doi:10.1145/3613905.3644053 - · DOI
  18. [20][20] Alex de Vries. 2023. The Growing Energy Footprint of Artificial Intelligence. Joule 7, 10 (Oct. 2023), 2191-2194. doi:10.1016/j.joule.2023.09.004 - · DOI · Paper page
  19. [21][21] Audrey Desjardins, Oscar Tomico, Andrés Lucero, Marta E. Cecchinato, and Carman Neustaedter. 2021. Introduction to the Special Issue on First-Person Methods in HCI. ACM Trans. Comput.-Hum. Interact. 28, 6 (Dec. 2021), 37:1-37:12. doi:10.1145/3492342 - · DOI
  20. [22][22] Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E. Gonzalez. 2024. VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models. https://openreview.net/forum?id=acxHV6werE -
  21. [23][23] Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. EtiCor: Corpus for Analyzing LLMs for Etiquettes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6921-6931. doi:10.18653/v1/2023.emnlp-main.428 - · DOI
  22. [24][24] Carolyn S Ellis and Arthur Bochner. 2000. Autoethnography, Personal Narrative, Reflexivity: Researcher as Subject. In Handbook of Qualitative Research (2nd ed.), Norman K. Denzin and Yvonna S. Lincoln (Eds.). SAGE Publications, Inc. -
  23. [26][26] Kathleen Fraser and Svetlana Kiritchenko. 2024. Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , Yvette Graham and Matthew Purver (Eds.). Association for Computational Linguistics, St. Julian's, Malta, 690-713. doi:10.18653/v1/2024.eacl-long.41 - · DOI
  24. [27][27] Prakhar Ganesh, Afaf Taik, and Golnoosh Farnadi. 2025. The Curious Case of Arbitrariness in Machine Learning. doi:10.48550/arXiv.2501.14959 arXiv:2501.14959 [cs]. - · DOI
  25. [28][28] Xiao Ge, Chunchen Xu, Daigo Misaki, Hazel Rose Markus, and Jeanne L Tsai. 2024. How Culture Shapes What People Want From AI. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24) . Association for Computing Machinery, New York, NY, USA, 1-15. doi:10.1145/3613904.3642660 - · DOI
  26. [29][29] Clifford Geertz. 2008. Thick Description: Toward an Interpretive Theory of Culture. In The Cultural Geography Reader (1st ed.), Timothy Oakes and Patricia L. Price (Eds.). Routledge, London. doi:10.4324/9780203931950 - · DOI
  27. [31][31] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. 2023. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. Advances in Neural Information Processing Systems 36 (Dec. 2023), 52132-52152. https://proceedings.neurips.cc/paper\_files/paper/2023/hash/ a3bf71c7c63f0c3bcb7ff67c67b1e7b1-Abstract-Datasets\_and\_Benchmarks.html -
  28. [32][32] Sourojit Ghosh and Aylin Caliskan. 2023. 'Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6971-6985. doi:10.18653/v1/2023.findings-emnlp.465 - · DOI
  29. [33][33] Sourojit Ghosh, Pranav Narayanan Venkit, Sanjana Gautam, Shomir Wilson, and Aylin Caliskan. 2024. Do Generative AI Models Output Harm While Representing Non-Western Cultures: Evidence from A Community-Centered Approach. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7, 1 (Oct. 2024), 476-489. doi:10.1609/aies.v7i1.31651 - · DOI
  30. [35][35] Mohammad Hajiesmaili, Shaolei Ren, Ramesh Sitaraman, and Adam Wierman. 2025. Toward Environmentally Equitable AI. Commun. ACM 68, 7 (June 2025), 70-73. doi:10.1145/3725980 - · DOI
  31. [36][36] Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. 2024. Towards Geographic Inclusion in the Evaluation of Text-to-Image Models. In The 2024 ACM Conference on Fairness Accountability and Transparency . ACM, Rio de Janeiro Brazil, 585-601. doi:10.1145/3630106.3658927 - · DOI · Paper page
  32. [37][37] Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. 2023. DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity. (2023). doi:10.48550/ARXIV.2308.06198 - · DOI
  33. [38][38] Noor Hammad, C. Ailie Fraser, Erik Harpstead, Jessica Hammer, and Mira Dontcheva. 2025. 'It's More of a Vibe I'm Going for': Designing Text-toMusic Generation Interfaces for Video Creators. In Proceedings of the 2025 ACM Designing Interactive Systems Conference (DIS '25) . Association for Computing Machinery, New York, NY, USA, 2738-2754. doi:10.1145/3715336.3735814 - · DOI
  34. [39][39] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. https://openreview.net/forum?id=d7KBjmI3GmQ -
  35. [40][40] Geert Hofstede. 1984. Culture's Consequences: International Differences in Work-Related Values . SAGE. -
  36. [41][41] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2I-CompBench: a comprehensive benchmark for open-world compositional text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23) . Curran Associates Inc., Red Hook, NY, USA, 78723-78747. https://proceedings.neurips.cc/paper\_files/paper/2023/file/f8ad010cdd9143dbb0e9308c093aff24-Paper-Datasets\_and\_Benchmarks.pdf -
  37. [42][42] Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, and Siwei Lyu. 2024. Exposing Text-Image Inconsistency Using Diffusion Models. In The Twelfth International Conference on Learning Representations . -
  38. [43][43] Nanna Inie, Jeanette Falk, and Raghavendra Selvan. 2025. How CO2STLY Is CHI? The Carbon Footprint of Generative AI in HCI Research and What We Should Do About It. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, 1-29. doi:10.1145/3706598.3714227 - · DOI
  39. [44][44] Justin Jager, Diane L. Putnick, and Marc H. Bornstein. 2017. More than Just Convenient: The Scientific Merits of Homogeneous Convenience Samples. Monogr Soc Res Child Dev 82, 2 (June 2017), 13-30. doi:10.1111/mono.12296 - · DOI · Paper page
  40. [45][45] Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan K Reddy, and Sunipa Dev. 2024. ViSAGe: A global-scale analysis of visual stereotypes in text-to-image generation. arXiv preprint arXiv:2401.06310 (2024). - · arXiv
  41. [46][46] Nithish Kannen, Arif Ahmad, Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, and Shachi Dave. 2024. Beyond aesthetics: Cultural competence in text-to-image models. arXiv preprint arXiv:2407.06863 (2024). - · arXiv
  42. [47][47] Elise Karinshak, Amanda Hu, Kewen Kong, Vishwanatha Rao, Jingren Wang, Jindong Wang, and Yi Zeng. 2024. LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output. arXiv:2411.06032 [cs] doi:10.48550/arXiv.2411.06032 - · DOI
  43. [48][48] Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig. 2024. An image speaks a thousand words, but can everyone listen? On translating images for cultural relevance. CoRR (Jan. 2024). https://openreview.net/forum?id=sZKojN0i9r -
  44. [49][49] Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2024) . https://doi.org/10.48550/arXiv.2404.16019 - · DOI
  45. [50][50] Jon Kleinberg and Manish Raghavan. 2021. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences 118, 22 (June 2021), e2018340118. doi:10.1073/pnas.2018340118 Publisher: Proceedings of the National Academy of Sciences. - · DOI · Paper page
  46. [51][51] Klaus Krippendorff. 2004. Reliability in Content Analysis. Human Communication Research 30, 3 (2004), 411-433. doi:10.1111/j.1468-2958.2004. tb00738.x \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1468-2958.2004.tb00738.x. - · DOI · Paper page
  47. [52][52] Tzu-Sheng Kuo, Aaron Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. 2024. Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-24. doi:10.1145/3613904.3642278 arXiv:2402.14147 [cs]. - · DOI
  48. [53][53] Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Emily Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. GenAI-Bench: A Holistic Benchmark for Compositional Text-to-Visual Generation. In Synthetic Data for Computer Vision Workshop @ CVPR 2024 . https: //openreview.net/forum?id=hJm7qnW3ym -
  49. [54][54] Huihan Li, Liwei Jiang, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024. CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting. https://openreview.net/forum?id=DbsLm2KAqP -
  50. [55][55] Bingshuai Liu, Longyue Wang, Chenyang Lyu, Yong Zhang, Jinsong Su, Shuming Shi, and Zhaopeng Tu. 2023. On the Cultural Gap in Text-to-Image Generation. arXiv:2307.02971 [cs] doi:10.48550/arXiv.2307.02971 - · DOI
  51. [59][59] Keziah Naggita, Julienne LaChance, and Alice Xiang. 2023. Flickr Africa: Examining Geo-Diversity in Large-Scale, Human-Centric Visual Data. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES '23) . Association for Computing Machinery, New York, NY, USA, 520-530. doi:10.1145/3600211.3604659 - · DOI
  52. [60][60] Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having Beer after Prayer? Measuring Cultural Bias in Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 16366-16393. doi:10.18653/v1/2024.acl-long.862 - · DOI · Paper page
  53. [61][61] Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. 2024. Benchmarking Vision Language Models for Cultural Understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 5769-5790. doi:10.18653/v1/2024.emnlp-main.329 - · DOI
  54. [63][63] Sophie J. Nightingale and Hany Farid. 2022. AI-synthesized faces are indistinguishable from real faces and more trustworthy. Proceedings of the National Academy of Sciences 119, 8 (2022), e2120481119. doi:10.1073/pnas.2120481119 - · DOI · Paper page
  55. [64][64] Claudia Núñez-Pacheco and Anton Poikolainen Rosén. 2024. Articulating Felt Senses for More-Than-Human Design: A Viewpoint for Noticing. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS '24) . Association for Computing Machinery, New York, NY, USA, 1029-1043. doi:10.1145/3643834.3661554 - · DOI
  56. [68][68] Ye Sul Park. 2024. White Default: Examining Racialized Biases Behind AI-Generated Images. Art Education 77, 4 (July 2024), 36-45. doi:10.1080/ 00043125.2024.2330340 Publisher: Routledge \_eprint: https://doi.org/10.1080/00043125.2024.2330340. - · DOI
  57. [69][69] Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, and Ibrahim Alabdulmohsin. 2025. No filter: cultural and socioeconomic diversity in contrastive vision-language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NIPS '24, Vol. 37) . Curran Associates Inc., Red Hook, NY, USA, 106474-106496. -
  58. [70][70] Shah Prerak. 2024. Addressing Bias in Text-to-Image Generation: A Review of Mitigation Methods. In 2024 Third International Conference on Smart Technologies and Systems for Next Generation Computing (ICSTSN) . 1-6. doi:10.1109/ICSTSN61422.2024.10671230 - · DOI
  59. [71][71] Rida Qadri, Mark Diaz, Ding Wang, and Michael Madaio. 2025. The Case for 'Thick Evaluations' of Cultural Representation in AI. arXiv preprint arXiv:2503.19075 (2025). - · arXiv
  60. [72][72] Rida Qadri, Renee Shelby, Cynthia L. Bennett, and Remi Denton. 2023. AI's Regimes of Representation: A Community-centered Study of Text-toImage Models in South Asia. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 506-517. doi:10.1145/3593013.3594016 - · DOI · Paper page
  61. [73][73] Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the Everything in the Whole Wide World Benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (Dec. 2021). https://datasets-benchmarksproceedings.neurips.cc/paper/2021/hash/084b6fbb10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html -
  62. [74][74] Gerald Roche. 2019. Articulating Language Oppression: Colonialism, Coloniality and the Erasure of Tibet's Minority Languages. Patterns of Prejudice 53, 5 (Oct. 2019), 487-514. doi:10.1080/0031322X.2019.1662074 - · DOI
  63. [75][75] David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji. 2024. CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark. arXiv:2406.05967 [cs] doi:10.48550/arXiv.2406.05967 - · DOI
  64. [77][77] Pola Schwöbel, Jacek Golebiowski, Michele Donini, Cedric Archambeau, and Danish Pruthi. 2023. Geographical Erasure in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2023 . Association for Computational Linguistics, Singapore, 12310-12324. doi:10.18653/v1/2023.findings-emnlp.823 - · DOI
  65. [78][78] Ali Akbar Septiandri, Marios Constantinides, Mohammad Tahaei, and Daniele Quercia. 2023. WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic Is FAccT?. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 160-171. doi:10.1145/3593013.3593985 - · DOI
  66. [79][79] Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N'Mah Yilla, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023. Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES '23) . Association for Computing Machinery, New York, NY, USA, 723-741. doi:10.1145/3600211.3604673 - · DOI · Paper page
  67. [80][80] Philip Wootaek Shin, Jihyun Janice Ahn, Wenpeng Yin, Jack Sampson, and Vijaykrishnan Narayanan. 2024. Can Prompt Modifiers Control Bias? A Comparative Analysis of Text-to-Image Generative Models. doi:10.48550/arXiv.2406.05602 arXiv:2406.05602 [cs]. - · DOI
  68. [81][81] C. Spearman. 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology 15, 1 (1904), 72-101. doi:10.2307/1412159 Publisher: University of Illinois Press. - · DOI · Paper page
  69. [83][83] Kyle Steinfeld. 2023. Clever Little Tricks: A Socio-Technical History of Text-to-Image Generative Models. International Journal of Architectural Computing 21, 2 (June 2023), 211-241. doi:10.1177/14780771231168230 - · DOI
  70. [85][85] Alina Valyaeva. 2023. AI Image Statistics for 2024: How Much Content Was Created by AI. https://journal.everypixel.com/ai-image-statistics -
  71. [86][86] Marion Walton. 2010. Social Distance, Mobility and Place: Global and Intimate Genres in Geo-Tagged Photographs of Guguletu, South Africa. In Proceedings of the 8th ACM Conference on Designing Interactive Systems (DIS '10) . Association for Computing Machinery, New York, NY, USA, 35-38. doi:10.1145/1858171.1858178 - · DOI
  72. [87][87] Marion Walton. 2014. Pavement Internet: Mobile Media Economies and Ecologies for Young People in South Africa. In The Routledge Companion to Mobile Media . Routledge, 450-461. https://www.taylorfrancis.com/routledge-handbooks/?context=rho -
  73. [88][88] Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. 2024. Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation. doi:10.48550/arXiv.2404.01030 arXiv:2404.01030 [cs]. - · DOI
  74. [89][89] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , Tal Linzen, Grzegorz Chrupała, and Afra Alishahi (Eds.). Association for Computational Linguistics, Brussels, Belgium, 353-355. doi:10.18653/v1/W18-5446 - · DOI · Paper page
  75. [90][90] Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 1404-1422. doi:10.18653/v1/2024.naacl-long.77 - · DOI
  76. [92][92] Katherine Yoon. 2022. Beneath the Surface: The Struggles of Dismantling Lookism in Looks-Obsessed Korea. Embodied: The Stanford Undergraduate Journal of Feminist, Gender, and Sexuality Studies 1, 1 (June 2022). https://ojs.stanford.edu/ojs/index.php/sjfgss/article/view/2118 -
  77. [93][93] Giulia Zaino, Carmine Tommaso Recchiuto, and Antonio Sgorbissa. 2023. Culture-to-Culture Image Translation and User Evaluation. arXiv:2201.01565 [cs] doi:10.48550/arXiv.2201.01565 - · DOI
  78. [94][94] Lili Zhang, Xi Liao, Zaijia Yang, Baihang Gao, Chunjie Wang, Qiuling Yang, and Deshun Li. 2024. Partiality and Misconception: Investigating Cultural Representativeness in Text-to-Image Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1-25.

Cited by

Sorted by citations

Authors
Kiden, S., Peter, O., Reyes-Cruz, G., Klyshbekova, M., Choi, S., Bergin, A. G., Waheed, M., Eke, D., Azim, T., Ramchurn, S., Stein, S., Vallejos, E. P., Devlin, K., & Fischer, J. E.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2510.27361
arXiv
2510.27361
OpenAlex
W4415923611
Cited by
0
References
0
Type
other
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • ePrints Soton (University of Southampton) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
text-to-image models, cultural sensitivity, cultural misrepresentation, community-based assessment, co-creation workshops, first-person methods, training data bias, minority groups, evaluation methodology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review