CoChat
Sign Up
Cited by 15Natural Language Processing TechniquesEthics and Social Impacts of AILexicography and Language Studies

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Tackling Language Modelling Bias in Support of Linguistic Diversity

Open access
Bella, Gábor, Helm, Paula, Koch, Gertraud, Giunchiglia, Fausto·2024·The 2024 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3630106.3658925
Open this paper in CoChat — free account

Abstract

Current AI-based language technologies—language models, machine translation systems, multilingual dictionaries and corpora—are known to focus on the world’s 2–3% most widely spoken languages. Research efforts of the past decade have attempted to expand this coverage to ‘under-resourced languages.’ The goal of our paper is to bring attention to a corollary phenomenon that we call language modelling bias: multilingual language processing systems often exhibit a hardwired, yet usually involuntary and hidden representational preference towards certain languages. We define language modelling bias as uneven per-language performance under similar test conditions. We show that bias stems not only from technology but also from ethically problematic research and development methodologies that disregard the needs of language communities. Moving towards diversity-aware alternatives, we present an initiative that aims at reducing language modelling bias within lexical resources through both technology design and methodology, based on an eye-level collaboration with local communities.

Overview

Integrity signals

Registry

Crossref · 10.1145/3630106.3658925

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. 10.18653/v1/D16-1250 · DOI
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization Jade Goldstein Alon Lavie Chin-Yew Lin and Clare Voss (Eds.). Association for Computational Linguistics Ann Arbor Michigan 65–72. https://aclanthology.org/W05-0909
  3. Khuyagbaatar Batsuren, Amarsanaa Ganbold, Altangerel Chagnaa, and Fausto Giunchiglia. 2019. Building the mongolian wordnet. In Proceedings of the 10th global WordNet conference. 238–244.
  4. Khuyagbaatar Batsuren Ekaterina Vylomova Verna Dankers Tsetsuukhei Delgerbaatar Omri Uzan Yuval Pinter and Gábor Bella. 2024. Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge. arxiv:2404.13292 [cs.CL] · arXiv
  5. David Beer. 2017. The Social Power of Algorithms. Information, Communication & Society 20, 1 (2017).
  6. Gábor Bella, Khuyagbaatar Batsuren, Temuulen Khishigsuren, and Fausto Giunchiglia. 2022. Linguistic Diversity and Bias in Online Dictionaries. University of Bayreuth African Studies Online (2022), 173.
  7. 10.18653/v1/2022.acl-demo.15 · DOI
  8. Gábor Bella, Fiona McNeill, Rody Gorman, Caoimhín Ó Donnaíle, Kirsty MacDonald, Yamini Chandrashekar, Abed Alhakim Freihat, and Fausto Giunchiglia. 2020. A major wordnet for a minority language: Scottish Gaelic. In Proceedings of The 12th Language Resources and Evaluation Conference. 2812–2818.
  9. Emily M Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology 6 (2011).
  10. Pushpak Bhattacharyya. 2010. IndoWordNet. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10).
  11. 10.18653/v1/2020.coling-main.313 · DOI · Paper page
  12. 10.18653/v1/2022.acl-long.539 · DOI
  13. Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual wordnet. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1352–1362.
  14. Francis Bond, Luis Morgado da Costa, Michael Wayne Goodman, John Philip McCrae, and Ahti Lohk. 2020. Some Issues with Building a Multilingual Wordnet. In Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 3189–3197. https://aclanthology.org/2020.lrec-1.390
  15. Nandu Chandran Nair, Rajendran S. Velayuthan, Yamini Chandrashekar, Gábor Bella, and Fausto Giunchiglia. 2022. IndoUKC: A Concept-Centered Indian Multilingual Lexical Resource. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 2833–2840. https://aclanthology.org/2022.lrec-1.303
  16. 10.1145/11952.107 · DOI
  17. Abed Alhakim Freihat, Hadi Khalilia, Gábor Bella, and Fausto Giunchiglia. 2024. Advancing the Arabic WordNet: Elevating Content Quality. In 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), LREC-COLING.
  18. Batya Friedman, Peter Kahn, and Alan Borning. 2002. Value sensitive design: Theory and methods. University of Washington technical report 2, 8 (2002).
  19. Lisa Gitelman. 2013. “Raw Data” Is an Oxymoron. MIT Press, Cambridge, MA, USA.
  20. Fausto Giunchiglia Khuyagbaatar Batsuren and Gabor Bella. 2017. Understanding and Exploiting Language Diversity.. In IJCAI. 4009–4017. · DOI
  21. Fausto Giunchiglia, Gábor Bella, Nandu C Nair, Yang Chi, and Hao Xu. 2023. Representing interlingual meaning in lexical databases. Artificial Intelligence Review (2023), 1–17.
  22. 10.2307/410659 · DOI
  23. 10.1080/00437956.1956.11659599 · DOI
  24. 10.1007/BF01064504 · DOI
  25. 10.1007/s10676-023-09742-6 · DOI · Paper page
  26. 10.1145/3514094.3534149 · DOI
  27. 10.18653/v1/2023.acl-long.396 · DOI
  28. 10.1145/1753326.1753522 · DOI · Paper page
  29. 10.18653/v1/2020.acl-main.560 · DOI · Paper page
  30. Paul Kay, Brent Berlin, Luisa Maffi, William R Merrifield, and Richard Cook. 2009. The world color survey. CSLI Publications Stanford, CA.
  31. Hadi Khalilia, Gábor Bella, Abed Alhakim Freihat, Shandy Darma, and Fausto Giunchiglia. 2023. Lexical diversity in kinship across languages and dialects. Frontiers in Psychology 14 (2023).
  32. Temuulen Khishigsuren, Gábor Bella, Khuyagbaatar Batsuren, Abed Alhakim Freihat, Nandu Chandran Nair, Amarsanaa Ganbold, Hadi Khalilia, Yamini Chandrashekar, and Fausto Giunchiglia. 2022. Using Linguistic Typology to Enrich Multilingual Lexicons: the Case of Lexical Gaps in Kinship. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2798–2807.
  33. 10.18653/v1/2022.findings-acl.44 · DOI
  34. 10.18653/v1/P19-1491 · DOI · Paper page
  35. George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
  36. 10.2307/3772751 · DOI
  37. Roberto Navigli and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics. 216–225.
  38. 10.18653/v1/2020.findings-emnlp.195 · DOI
  39. Nanjala Nyabola. 2018. Digital Democracy, Analogue Politics: How the Internet Era is Transforming Politics in Kenya. Zed Books, London.
  40. 10.18653/v1/N19-1356 · DOI · Paper page
  41. Jan Rijkhoff, Dik Bakker, Kees Hengeveld, and Peter Kahrel. 1993. A method of language sampling. Studies in Language. International Journal sponsored by the Foundation“Foundations of Language” 17, 1 (1993), 169–203.
  42. 10.18653/v1/2021.acl-long.243 · DOI · Paper page
  43. 10.18653/v1/2022.acl-short.82 · DOI
  44. NLLB Team Marta R. Costa-jussà James Cross OnurÇelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Janice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Francisco Guzmán Philipp Koehn Alexandre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk and Jeff Wang. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arxiv:2207.04672 [cs.CL] · arXiv
  45. 10.18653/v1/2021.eacl-main.188 · DOI
  46. 10.1017/S000397561200015X · DOI
  47. 10.18653/v1/2021.acl-long.38 · DOI
  48. Iris Marion Young. 1990. Justice and the Politics of Difference. Princeton University Press. Google-Books-ID: Q6keKguPrsAC.
  49. Michael J. Zimmerman and Ben Bradley. 2019. Intrinsic vs. Extrinsic Value. In The Stanford Encyclopedia of Philosophy (Spring 2019 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.

Cited by

Sorted by citations

Metadata

Authors
Bella, G., Helm, P., Koch, G., & Giunchiglia, F.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • University of Amsterdam (NL)
  • Universität Hamburg (DE)
  • University of Trento (IT)
ORCID
Venue
The 2024 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2024-06-03
Pages
562-572
DOI
10.1145/3630106.3658925
OpenAlex
W4396600241
Cited by
15
References
5
Citations by year
2026: 4 · 2025: 10 · 2024: 1
FWCI
4.43
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
  • HAL (submittedVersion, cc-by-nc-nd)
  • Iris (University of Trento) (submittedVersion)
Funders
Data Science Centre, University of Amsterdam
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Arts and Humanities › Language and Linguistics › Lexicography and Language Studies
Keywords
linguistic diversity, low-resource languages, multilingual language models, lexical resources, multilingual dictionary, community collaboration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review