Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 4

The DOI matches its DataCite record.

Large Language Models are Geographically Biased

Manvi, Rohin, Khanna, Samar, Burke, Marshall, Lobell, David, Ermon, Stefano·2024·arXiv
DOI 10.48550/arxiv.2402.02680arXiv 2402.02680
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $ρ$ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's $ρ$ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.02680

Cites

Statistics

Figures

Figure 1 The mean rank plots illustrate agreement across LLM predictions, with areas of green and red highlighting regions consistently rated higher or lower respectively. For objective topics, the maps demonstrate the zero-shot geographic knowledge of LLMs. The sensitive subjective topics reveal agreement that indicates strong socioeconomic biases. The geographically independent topics serve as the control.
Figure 1 The mean rank plots illustrate agreement across LLM predictions, with areas of green and red highlighting regions consistently rated higher or lower respectively. For objective topics, the maps demonstrate the zero-shot geographic knowledge of LLMs. The sensitive subjective topics reveal agreement that indicates strong socioeconomic biases. The geographically independent topics serve as the control.CC-BY · From the PDF
Figure 3 Zero-shot GPT-4 Turbo comparison with ground truth.
Figure 3 Zero-shot GPT-4 Turbo comparison with ground truth.CC-BY · From the PDF
Figure 4 Common biases on important objective topics. This is shown with mean rank error where red points indicate common overestimates in rank and blue points indicate common underestimates.
Figure 4 Common biases on important objective topics. This is shown with mean rank error where red points indicate common overestimates in rank and blue points indicate common underestimates.CC-BY · From the PDF
Figure 5 Demonstration of bias scores for Average Likability of Residents. The anchoring bias distribution is socioeconomic conditions.
Figure 5 Demonstration of bias scores for Average Likability of Residents. The anchoring bias distribution is socioeconomic conditions.CC-BY · From the PDF
Figure 6 Geographic biases of GPT-3.5 Turbo in the Bay Area, California with respect to the 'Average Intelligence of Resident'. Red corresponds to a lower rating and vice-versa for green.
Figure 6 Geographic biases of GPT-3.5 Turbo in the Bay Area, California with respect to the 'Average Intelligence of Resident'. Red corresponds to a lower rating and vice-versa for green.CC-BY · From the PDF
Figure 7 Enlarged version of Figure 4 with dark background.
Figure 7 Enlarged version of Figure 4 with dark background.CC-BY · From the PDF
Image 7
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 55

  1. Abid, A., Farooqi, M., and Zou, J. Y. Persistent anti- muslim bias in large language models. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021. URL https://api.semanticscholar. org/CorpusID:231603388.
  2. Ahn, J. and Oh, A. H. Mitigating language- dependent ethnic bias in bert. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://api.semanticscholar. org/CorpusID:237491723.
  3. Bartl, M., Nissim, M., and Gatt, A. Unmask- ing contextual stereotypes: Measuring and mitigat- ing bert's gender bias. ArXiv, abs/2010.14534, 2020. URL https://api.semanticscholar. org/CorpusID:225094152. · arXiv
  4. Besse, P. C., del Barrio, E., Gordaliza, P., Loubes, J.-M., and Risser, L. A survey of bias in ma- chine learning through the prism of statistical par- ity. The American Statistician, 76:188 -198, 2020. URL https://api.semanticscholar. org/CorpusID:214727646.
  5. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. · arXiv
  6. Cao, Y. T. and Daumé, H. Toward gender-inclusive coreference resolution. ArXiv, abs/1910.13913, 2019. URL https://api.semanticscholar. org/CorpusID:204961553. · arXiv
  7. Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5 2:153-163, 2016. URL https://api. semanticscholar.org/CorpusID:1443041.
  8. Dash, D., Thapa, R., Banda, J., Swaminathan, A., Cheatham, M., Kashyap, M., Kotecha, N., Chen, J. H., Gom- bar, S., Downing, L., Pedreira, R. A., Goh, E., Ar- naout, A., Morris, G. K., Magon, H., Lungren, M. P., Horvitz, E., and Shah, N. H. Evaluation of gpt- 3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. ArXiv, abs/2304.13714, 2023. URL https://api.semanticscholar. org/CorpusID:258331653. · arXiv
  9. de Vassimon Manela, D., Errington, D., Fisher, T., van Breugel, B., and Minervini, P. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. ArXiv, abs/2101.09688, 2021. URL https://api.semanticscholar. org/CorpusID:231698886. · arXiv
  10. Del'etang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Li, W. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. 2023. URL https://api.semanticscholar. org/CorpusID:262054258.
  11. Deng, C., Zhang, T., He, Z., Chen, Q., Shi, Y., Zhou, L., Fu, L., Zhang, W., Wang, X., Zhou, C., et al. Learning a foundation language model for geoscience knowledge understanding and utilization. arXiv preprint arXiv:2306.05064, 2023. · arXiv
  12. Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., and Gupta, R. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, 2021. URL https://api.semanticscholar. org/CorpusID:231719337.
  13. Diaz, M., Johnson, I. L., Lazar, A., Piper, A. M., and Ger- gle, D. Addressing age-related bias in sentiment analysis. Proceedings of the 2018 CHI Conference on Human Fac- tors in Computing Systems, 2018. URL https://api. semanticscholar.org/CorpusID:3272048.
  14. Dodge, J., Marasovic, A., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Document- ing large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://api.semanticscholar. org/CorpusID:237568724.
  15. Du, Y., Fang, Q., and Nguyen, D. Assessing the reliability of word embedding gender bias mea- sures. ArXiv, abs/2109.04732, 2021. URL https: //api.semanticscholar.org/CorpusID: 237485538. · arXiv
  16. Elvidge, C. D., Baugh, K. E., Zhizhin, M. N., Hsu, F.- C., and Ghosh, T. Viirs night-time lights. Interna- tional Journal of Remote Sensing, 38:5860 -5879, 2017. URL https://api.semanticscholar. org/CorpusID:264155349.
  17. Florczyk, A. J., Corbane, C., Ehrlich, D., Freire, S., Kemper, T., Maffenini, L., Melchiorri, M., Pesaresi, M., Politis, P., Schiavina, M., Sabo, F., and Zanchetta, L. GHSL Data Package 2019. Number JRC 117104 in EUR 29788 EN. Publications Office of the European Union, Luxembourg, 2019. ISBN 978-92-76-13186-1. doi: 10.2760/290498. · DOI
  18. Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., and Ahmed, N. Bias and fairness in large lan- guage models: A survey. ArXiv, abs/2309.00770, 2023. URL https://api.semanticscholar. org/CorpusID:261530629. · arXiv
  19. Garg, N., Schiebinger, L., Jurafsky, D., and Zou, J. Y. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sci- ences, 115:E3635 -E3644, 2017. URL https://api. semanticscholar.org/CorpusID:4930886.
  20. Garimella, A., Amarnath, A., Kumar, K., Yalla, A. P., Natarajan, A., Chhaya, N., and Srinivasan, B. V. He is very intelligent, she is very beautiful? on mitigating social biases in language modelling and generation. In Findings, 2021. URL https://api.semanticscholar. org/CorpusID:236477795.
  21. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings, 2020. URL https://api.semanticscholar. org/CorpusID:221878771.
  22. Google. Gemini: A family of highly capable mul- timodal models. ArXiv, abs/2312.11805, 2023. URL https://api.semanticscholar.org/ CorpusID:266361876. · arXiv
  23. Gupta, V., Venkit, P. N., Wilson, S., and Passon- neau, R. Survey on sociodemographic bias in nat- ural language processing. ArXiv, abs/2306.08158, 2023. URL https://api.semanticscholar. org/CorpusID:259164882. · arXiv
  24. Hardt, M., Price, E., and Srebro, N. Equality of oppor- tunity in supervised learning. ArXiv, abs/1610.02413, 2016. URL https://api.semanticscholar. org/CorpusID:7567061. · arXiv
  25. Huang, Y., Zhang, Q., Yu, P. S., and Sun, L. Trust- gpt: A benchmark for trustworthy and responsi- ble large language models. ArXiv, abs/2306.11507, 2023. URL https://api.semanticscholar. org/CorpusID:259202452. · arXiv
  26. Jean, N., Burke, M., Xie, S. M., Davis, W. M., Lobell, D., and Ermon, S. Combining satellite imagery and ma- chine learning to predict poverty. Science, 353:790 -794, 2016. URL https://api.semanticscholar. org/CorpusID:16154009.
  27. Kamiran, F. and Calders, T. Data preprocess- ing techniques for classification without discrimina- tion. Knowledge and Information Systems, 33:1-33, 2012. URL https://api.semanticscholar. org/CorpusID:14637938.
  28. Karger, D. N., Conrad, O., Böhner, J., Kawohl, T., Kreft, H., Soria-Auza, R. W., Zimmermann, N. E., Linder, H. P., and Kessler, M. Data from: Climatologies at high resolution for the earth's land surface areas. EnviDat, 2018.
  29. Krieg, K., Parada-Cabaleiro, E., Medicus, G., Lesota, O., Schedl, M., and Rekabsaz, N. Grep-biasir: A dataset for investigating gender representation bias in informa- tion retrieval results. Proceedings of the 2023 Confer- ence on Human Information Interaction and Retrieval, 2022. URL https://api.semanticscholar. org/CorpusID:246035444.
  30. Li, T., Khashabi, D., Khot, T., Sabharwal, A., and Srikumar, V. Unqovering stereotypical bi- ases via underspecified questions. In Findings, 2020. URL https://api.semanticscholar. org/CorpusID:222141056.
  31. Liang, P. P., Wu, C., Morency, L.-P., and Salakhutdinov, R. Towards understanding and mitigating social biases in language models. In International Con- ference on Machine Learning, 2021. URL https: //api.semanticscholar.org/CorpusID: 235623756.
  32. Mahabadi, R. K., Belinkov, Y., and Henderson, J. End-to-end bias mitigation by modelling biases in corpora. In Annual Meeting of the Association for Computational Linguistics, 2019. URL https: //api.semanticscholar.org/CorpusID: 215191351.
  33. Mai, G., Huang, W., Sun, J., Song, S., Mishra, D., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. On the opportu- nities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304.06798, 2023. · arXiv
  34. Manvi, R., Khanna, S., Mai, G., Burke, M., Lobell, D. B., and Ermon, S. Geollm: Extracting geospatial knowledge from large language models. ArXiv, abs/2310.06213, 2023. URL https://api.semanticscholar. org/CorpusID:263831484. · arXiv
  35. Manzini, T., Lim, Y. C., Tsvetkov, Y., and Black, A. W. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In North American Chap- ter of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar. org/CorpusID:102350941. Meta. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023. URL https: //api.semanticscholar.org/CorpusID: 259950998. · arXiv
  36. Mistral. Mixtral of experts. ArXiv, abs/2401.04088, 2024. URL https://api.semanticscholar. org/CorpusID:266844877. · arXiv
  37. Nadeem, M., Bethke, A., and Reddy, S. Stereoset: Measuring stereotypical bias in pretrained language models. In Annual Meeting of the Association for Computational Linguistics, 2020. URL https: //api.semanticscholar.org/CorpusID: 215828184.
  38. Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Conference on Empirical Methods in Natural Language Processing, 2020. URL https://api.semanticscholar. org/CorpusID:222090785.
  39. Nozza, D., Bianchi, F., and Hovy, D. Hon- est: Measuring hurtful sentence completion in language models. In North American Chapter of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar. org/CorpusID:235097294.
  40. OpenAI. Gpt-4 technical report. 2023a. URL https: //api.semanticscholar.org/CorpusID: 257532815.
  41. Park, J. H., Shin, J., and Fung, P. Reducing gender bias in abusive language detection. In Conference on Empirical Methods in Natural Language Processing, 2018. URL https://api.semanticscholar. org/CorpusID:52070035.
  42. Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. Bbq: A hand-built bias benchmark for ques- tion answering. In Findings, 2021. URL https: //api.semanticscholar.org/CorpusID: 239010011.
  43. Perez, A., Yeh, C., Azzari, G., Burke, M., Lobell, D., and Er- mon, S. Poverty prediction with public landsat 7 satellite imagery and machine learning. ArXiv, abs/1711.03654, 2017. URL https://api.semanticscholar. org/CorpusID:23748178. · arXiv
  44. Schick, T., Udupa, S., and Schütze, H. Self-diagnosis and self-debiasing: A proposal for reducing corpus- based bias in nlp. Transactions of the Associ- ation for Computational Linguistics, 9:1408-1424, 2021. URL https://api.semanticscholar. org/CorpusID:232075876.
  45. Smith, E. M., Hall, M., Kambadur, M., Presani, E., and Williams, A. "i'm sorry to hear that": Find- ing new biases in language models with a holis- tic descriptor dataset. In Conference on Em- pirical Methods in Natural Language Processing, 2022. URL https://api.semanticscholar. org/CorpusID:253224433.
  46. Tan, Y. C. and Celis, E. Assessing social and inter- sectional biases in contextualized word representa- tions. ArXiv, abs/1911.01485, 2019. URL https: //api.semanticscholar.org/CorpusID: 202781363. · arXiv
  47. Tatem, A. J. Worldpop, open data for spatial demogra- phy. Scientific Data, 4, 2017. URL https://api. semanticscholar.org/CorpusID:3544507.
  48. Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T., and Wilson, S. Unmasking nationality bias: A study of human perception of nationalities in ai-generated articles. Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 2023. URL https://api.semanticscholar. org/CorpusID:260704383.
  49. Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., and Petrov, S. Measuring and reducing gendered correlations in pre-trained models. ArXiv, abs/2010.06032, 2020. URL https://api.semanticscholar. org/CorpusID:222310622. · arXiv
  50. WorldPop and CIESIN, C. U. Global high res- olution population denominators project -funded by the bill and melinda gates foundation, 2018. URL https://dx.doi.org/10.5258/SOTON/ WP00647. www.worldpop.org -School of Geography and Environmental Science, University of Southampton; Department of Geography and Geosciences, University of Louisville; Departement de Geographie, Universite de Namur. · DOI
  51. Yeh, C., Perez, A., Driscoll, A., Azzari, G., Tang, Z., Lobell, D., Ermon, S., and Burke, M. Using publicly available satellite imagery and deep learning to understand eco- nomic well-being in africa. Nature Communications, 11, 2020. URL https://api.semanticscholar. org/CorpusID:218773287.
  52. Yeh, C., Meng, C., Wang, S., Driscoll, A., Rozi, E., Liu, P., Lee, J., Burke, M., Lobell, D., and Ermon, S. Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learn- ing. ArXiv, abs/2111.04724, 2021. URL https: //api.semanticscholar.org/CorpusID: 243847865. · arXiv
  53. Yogarajan, V., Dobbie, G., Keegan, T. T., and Neuwirth, R. J. Tackling bias in pre-trained language mod- els: Current trends and under-represented soci- eties. ArXiv, abs/2312.01509, 2023. URL https: //api.semanticscholar.org/CorpusID: 265609311. · arXiv
  54. Zhang, Y., Wei, C., Wu, S., He, Z., and Yu, W. Geogpt: Understanding and processing geospatial tasks through an autonomous gpt. arXiv preprint arXiv:2307.07930, 2023. · arXiv
  55. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J., and rong Wen, J. A sur- vey of large language models. ArXiv, abs/2303.18223, 2023. URL https://api.semanticscholar. org/CorpusID:257900969. · arXiv

Cited by

Metadata

Authors
Manvi, R., Khanna, S., Burke, M., Lobell, D., & Ermon, S.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2402.02680
arXiv
2402.02680
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.