CoChat
Sign Up
Cited by 13Ethics and Social Impacts of AICulinary Culture and TourismMobile Crowdsensing and Crowdsourcing

The World Wide recipe: A community-centred framework for fine-grained data collection and regional bias operationalisation

Open access
Magomere, Jabez, Ishida, Shu, Afonja, Tejumade, Salama, Aya, Kochin, Daniel, Foutse, Yuehgoh, Hamzaoui, Imane, Sefala, Raesetje +6 more·2025·Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3715275.3732019
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce the World Wide recipe, which sets forth a framework for culturally aware and participatory data collection, and the resultant regionally diverse World Wide Dishes evaluation dataset.We

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3715275.3732019

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Researchers have previously problematized anonymous crowdsourced data annotation approaches as being unspecific about who's worldviews and beliefs become manifest in the data.

    “Researchers have previously problematized anonymous crowdsourced data annotation approaches [31, 53, 65] as being unspecific about who's worldviews and beliefs become manifest in the data [11, 17, 51, 78, 90, 91].”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [78]

    Section 5 (‘Food for thought’; page unavailable) argues that who contributes to a dataset affects its representations and that community involvement helps align bias evaluation with local values. Section 2.1 describes collecting culturally grounded contributions. The paper does not specifically characterize anonymous crowdsourced annotation as leaving annotators’ worldviews or beliefs unspecified. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation or allow community members to self-select their participation, albeit with some limitations.

    “In more community-centric data collection or AI evaluation work, active efforts are made by researchers to solicit broad participation (e.g., across geographical regions [52, 68, 91] and through local research partners [89]), or allow community members to selfselect their participation, albeit with some limitations (e.g., adults, higher-education levels or topic expertise, English-language ability”

    From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [78]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics 9 (2021), 1116-1131.
  2. Grace Ataguba, Rock Ezekiel, James Daniel, Emeka Ogbuju, and Rita Orji. 2024. African foods for deep learning-based food recognition systems dataset. Data in Brief 53 (2024), 110092. doi:10.1016/j.dib.2024.110092 · DOI
  3. Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. 2022. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. arXiv preprint arXiv:2203.11933 (2022). · arXiv
  4. James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023. Improving im- age generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf 2, 3 (2023), 8.
  5. Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes. arXiv preprint arXiv:2403.05696 (2024). · arXiv
  6. Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1493-1504.
  7. Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022. Power to the people? Opportunities and challenges for participatory AI. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1-8.
  8. Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963 (2021). · arXiv
  9. Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13. Springer, 446-461.
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877-1901.
  11. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020). · arXiv
  12. Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. 2024. Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation. In ICLR.
  13. Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 52-59.
  14. Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. The participatory turn in ai design: Theoretical foundations and the current state of practice. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1-23.
  15. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248-255.
  16. Terrance DeVries, Adriana Romero, Luis Pineda, Graham W Taylor, and Michal Drozdzal. 2019. On the evaluation of conditional GANs. arXiv preprint arXiv:1907.08175 (2019). · arXiv
  17. Brooke Foucault Welles and Sandra González-Bailón. 2020. The Oxford Handbook of Networked Communication. Oxford University Press. doi:10.1093/oxfordhb/ 9780190460518.001.0001 · DOI
  18. Dan Friedman and Adji Bousso Dieng. 2023. The Vendi Score: A Diversity Evaluation Metric for Machine Learning. arXiv:2210.02410 [cs.LG] https://arxiv. org/abs/2210.02410 · arXiv
  19. William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The Dollar Street dataset: Images repre- senting the geographic and socioeconomic diversity of the world. Advances in Neural Information Processing Systems 35 (2022), 12979-12990.
  20. Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. 2024. DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity. arXiv:2308.06198 [cs.CV] https://arxiv.org/abs/2308.06198 · arXiv
  21. Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. 2022. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706 (2022). · arXiv
  22. Siobhan Mackenzie Hall, Samantha Dalal, Raesetje Sefala, Foutse Yuehgoh, Aisha Alaagib, Imane Hamzaoui, Shu Ishida, Jabez Magomere, Lauren Crais, Aya Salama, et al. 2025. The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes. arXiv preprint arXiv:2502.05961 (2025). · arXiv
  23. Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke, Aleksandar Shtedritski, and Hannah Rose Kirk. 2024. Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution. Advances in Neural Information Processing Systems 36 (2024).
  24. J Hardy. 2019. How the design of social technology fails rural America. DIS'19 Companion: Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion, 189-193.
  25. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021). · arXiv
  26. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).
  27. Rachel Hong, Tadayoshi Kohno, and Jamie Morgenstern. 2023. Evaluation of targeted dataset collection on racial equity in face recognition. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (Montréal, QC, Canada) (AIES '23). Association for Computing Machinery, New York, NY, USA, 531-541. doi:10.1145/3600211.3604662 · DOI
  28. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Kr- ishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20406-20417.
  29. Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. 2022. Un- derspecification in scene description-to-depiction tasks. arXiv preprint arXiv:2210.05815 (2022). · arXiv
  30. Henry Jenkins. 2009. Confronting the challenges of participatory culture: Media education for the 21st century. The MIT press.
  31. Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan K Reddy, and Sunipa Dev. 2024. Beyond the Surface: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation. arXiv preprint arXiv:2401.06310 (2024). · arXiv
  32. Nithish Kannen, Arif Ahmad, Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, and Shachi Dave. 2024. Beyond aesthetics: Cultural competence in text-to-image models. arXiv preprint arXiv:2407.06863 (2024). · arXiv
  33. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2019. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems 32 (2019).
  34. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved Baselines with Visual Instruction Tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. https://openreview.net/forum?id=yx3Hkx5ved
  35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024).
  36. Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, et al. 2024. Bridging the Data Provenance Gap Across Text, Speech and Video. arXiv preprint arXiv:2412.17847 (2024). · arXiv
  37. Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408 (2023). · arXiv
  38. Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. 2018. Recipe1M+: a dataset for learning cross-modal embeddings for cooking recipes and food images. arXiv preprint arXiv:1810.06553 (2018). · arXiv
  39. Erica Mitchell, Kevin G Crowston, and Carsten Oesterlund. 2018. Coordinat- ing advanced crowd work: Extending citizen science. In 51st Annual Hawai'i International Conference on System Sciences.
  40. Deirdre K Mulligan, Joshua A Kroll, Nitin Kohli, and Richmond Y Wong. 2019. This thing called fairness: Disciplinary confusion realizing a value in technology. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-36.
  41. Esther Mwema and Abeba Birhane. 2025. Undersea cables in Africa: The new frontiers of digital colonialism. First Monday 29, 4 (2025). doi:10.5210/fm.v29i4. 13637 [Accessed 13 January 2025]. · DOI
  42. Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. 2020. Reliable Fidelity and Diversity Metrics for Generative Models. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 7176-7185. https://proceedings.mlr.press/v119/naeem20a.html
  43. Quan Nguyen and Adji Bousso Dieng. 2024. Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design. arXiv:2405.02449 [stat.ML] https://arxiv.org/abs/2405.02449 · arXiv
  44. Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin'ichi Satoh. 2023. Toward Verifiable and Re- producible Human Evaluation for Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14277-14286.
  45. Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, and Ibrahim Alabdulmohsin. 2024. No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models. arXiv preprint arXiv:2405.13777 (2024). · arXiv
  46. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748-8763.
  47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1-67.
  48. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022), 3. · arXiv
  49. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967 (2024). · arXiv
  50. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263 (2023). · arXiv
  51. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems 35 (2022), 36479-36494.
  52. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems 35 (2022), 36479-36494.
  53. Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. 2018. Assessing generative models via precision and recall. Advances in neural information processing systems 31 (2018).
  54. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems 29 (2016).
  55. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35 (2022), 25278-25294.
  56. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021). · arXiv
  57. Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, Atlanta GA USA, 59-68. doi:10.1145/3287560.3287598 · DOI · Paper page
  58. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv preprint arXiv:1711.08536 (2017). · arXiv
  59. Shivalika Singh, Freddie Vargus, Daniel D'souza, Börje F Karlsson, Abinaya Ma- hendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O'Mahony, et al. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619 (2024). · arXiv
  60. Athula Sumathipala, Sisira Siribaddana, and Vikram Patel. 2004. Under- representation of developing countries in the research literature: ethical issues arising from a survey of five leading medical journals. BMC medical ethics 5 (2004), 1-6.
  61. Hanna Wallach, Meera Desai, Nicholas Pangakis, A Feder Cooper, Angelina Wang, Solon Barocas, Alexandra Chouldechova, Chad Atalla, Su Lin Blodgett, Emily Corvi, et al. 2024. Evaluating Generative AI Systems is a Social Science Measurement Challenge. arXiv preprint arXiv:2411.10939 (2024). · arXiv

Cited by

Sorted by citations

Authors
Magomere, J., Ishida, S., Afonja, T., Salama, A., Kochin, D., Foutse, Y., Hamzaoui, I., Sefala, R., Alaagib, A., Dalal, S., Marchegiani, B., Semenova, E., Crais, L., & Hall, S. M.
Affiliations
  • University of Oxford (GB)
  • Autodesk (United Kingdom) (GB)
  • Helmholtz Center for Information Security (DE)
  • Microsoft (United States) (US)
  • KmerAI, Toulon, France
  • École Nationale Supérieure d'Informatique (DZ)
  • McGill University (CA)
  • Independent researcher, Riyadh, Saudi Arabia
  • University of Colorado Boulder (US)
  • Imperial College London (GB)
ORCID
Venue
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2025-06-23
Pages
246-282
DOI
10.1145/3715275.3732019
OpenAlex
W4411550181
Cited by
13
References
2
Citations by year
2026: 9 · 2025: 4
FWCI
13.20
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
Funders
European Commission (101070617)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Life Sciences › Agricultural and Biological Sciences › Food Science › Culinary Culture and Tourism
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
regional bias, participatory data collection, regional variation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review