CoChat

Search bias quantification: investigating political bias in social media and web search

Open access
Kulshrestha, Juhi, Eslami, Motahhare, Messias, Johnnatan, Zafar, Muhammad Bilal, Ghosh, Saptarshi, Gummadi, Krishna P., Karahalios, Karrie · 2019 · Information Retrieval Journal
Cited by 103

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

Users frequently use search systems on the Web as well as online social media to learn about ongoing events and public opinion on personalities. Prior studies have shown that the top-ranked results returned by these search engines can shape user opinion about the topic (e.g., event or person) being searched. In case of polarizing topics like politics, where multiple competing perspectives exist, the political bias in the top search results can play a significant role in shaping public opinion towards (or away from) certain perspectives. Given the considerable impact that search bias can have on the user, we propose a generalizable search bias quantification framework that not only measures the political bias in ranked list output by the search system but also decouples the bias introduced by the different sources—input data and ranking system. We apply our framework to study the political bias in searches related to 2016 US Presidential primaries in Twitter social media search and find that both input data and ranking system matter in determining the final search output bias seen by the users. And finally, we use the framework to compare the relative bias for two popular search systems—Twitter social media search and Google web search—for queries related to politicians and political events. We end by discussing some potential solutions to signal the bias in the search results to make the users more aware of them.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10791-018-9341-2

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 20 references

Research works printed but not linked: 51

  1. Babaei, M., Kulshrestha, J., Chakraborty, A., Benevenuto, F., Gummadi, K. P., & Weller, A. (2018). Purple feed: Identifying high consensus news posts on social media. In Proceedings of the AAAI/ACM conference on artifical intelligence, ethics & society, AIES 2018, New Orleans, USA. · DOI
  2. Baron, D. P. (2006). Persistent media bias. Journal of Public Economics, 90(1–2), 1–36. · DOI · Paper page
  3. Bhattacharya, P., Ghosh, S., Kulshrestha, J., Mondal, M., Zafar, M. B., Ganguly, N., & Gummadi, K. P. (2014a). Deep twitter diving: Exploring topical groups in microblogs at scale. In Proceedings of the 17th ACM conference on computer supported cooperative work and social computing, CSCW ’14, pp. 197–210. ACM, New York, NY, USA. http://doi.acm.org/10.1145/2531602.2531636 · DOI
  4. Bhattacharya, P., Zafar, M. B., Ganguly, N., Ghosh, S., & Gummadi, K. P. (2014b). Inferring user interests in the twitter social network. In Proceedings of the 8th ACM conference on recommender systems, RecSys ’14, pp. 357–360. ACM, New York, NY, USA. https://doi.org/10.1145/2645710.2645765 . · DOI · Paper page
  5. Bond, R., & Messing, S. (2015). Quantifying social media’s political space: Estimating ideology from publicly revealed preferences on facebook. American Political Science Review, 109(01), 62–78. · DOI · Paper page
  6. Carney, D. R., Jost, J. T., Gosling, S. D., & Potter, J. (2008). The secret lives of liberals and conservatives: Personality profiles, interaction styles, and the things they leave behind. Political Psychology, 29(6), 807–840. https://doi.org/10.1111/j.1467-9221.2008.00668.x. . · DOI · Paper page
  7. Chen, L., Mislove, A., & Wilson, C. (2015). Peeking beneath the hood of UBER. In Proceedings of the 2015 ACM conference on internet measurement conference (pp. 495–508). ACM. · DOI · Paper page
  8. Cohen, R., & Ruths, D. (2013). Classifying political orientation on twitter: It’s not easy!. In Proceedings of AAAI international conference on web & social media, ICWSM 2013, Boston, USA.
  9. Coletto, M., Garimella, K., Gionis, A., & Lucchese, C. (2017). A motif-based approach for identifying controversy. arXiv:1703.05053 . · arXiv
  10. Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. In Proceedings on privacy enhancing technologies, http://arxiv.org/abs/1408.6491 . · arXiv
  11. Epstein, R., & Robertson, R. E. (2015). The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of of the National Academy of Sciences (PNAS), 112(33), E4512–E4521. · DOI · Paper page
  12. Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017a). Suppressing the search engine manipulation effect (SEME). Proceedings ACM Human–Computer Interaction, 1, 42:1–42:22. https://doi.org/10.1145/3134677 . · DOI · Paper page
  13. Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017b). Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human–Computer Interaction, 1(2), 452.
  14. Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015). I always assumed that i wasn’t really that close to [her]: Reasoning about invisible algorithms in news feeds. In Proceedings of the 33rd annual ACM conference on human factors in computing systems (pp. 153–162). ACM. · DOI · Paper page
  15. things can be worse than they appear": Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of AAAI international conference on web & social media, ICWSM 2017 (pp. 62-71). · DOI · Paper page
  16. Fang, A., Ounis, I., Habel, P., Macdonald, C., & Limsopatham, N. (2015). Topic-centric classification of twitter user’s political orientation. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’15 (pp. 791–794).
  17. Fortunato, S., Flammini, A., Menczer, F., & Vespignani, A. (2006). Topical interests and the mitigation of search engine bias. Proceedings of of the National Academy of Sciences (PNAS), 103(34), 12684–12689. · DOI · Paper page
  18. Gentzkow, M., & Shapiro, J. (2010). What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1), 35–71. · DOI · Paper page
  19. Ghosh, S., Sharma, N., Benevenuto, F., Ganguly, N., & Gummadi, K. (2012). Cognos: Crowdsourcing search for topic experts in microblogs. In Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’12 (pp. 575–590). ACM, New York, NY, USA. https://doi.org/10.1145/2348283.2348361 . · DOI · Paper page
  20. Golbeck, J., & Hansen, D. (2011). Computing political preference among Twitter followers. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1105–1108). ACM. · DOI
  21. Groseclose, T., & Milyo, J. (2005). A measure of media bias. The Quarterly Journal of Economics, 120(4), 1191–1237. · DOI · Paper page
  22. Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013). Measuring personalization of web search. In Proceedings of the 22nd international conference on world wide web, WWW ’13 (pp. 527–538). ACM, New York, NY, USA. https://doi.org/10.1145/2488388.2488435 . · DOI · Paper page
  23. Himelboim, I., McCreery, S., & Smith, M. (2013). Birds of a feather tweet together: Integrating network and content analyses to examine cross-ideology exposure on twitter. Journal of Computer-Mediated Communication, 18(2), 40–60. · DOI · Paper page
  24. Kliman-Silver, C., Hannak, A., Lazer, D., Wilson, C., & Mislove, A. (2015). Location, location, location: The impact of geolocation on web search personalization. In Proceedings of the 2015 internet measurement conference, IMC ’15 (pp. 121–127). ACM, New York, NY, USA. https://doi.org/10.1145/2815675.2815714 . · DOI
  25. Koutra, D., Bennett, P. N., & Horvitz, E. (2015). Events and controversies: Influences of a shocking news event on information seeking. In Proceedings of the 24th international conference on world wide web, WWW ’15, International world wide web conferences steering committee, Republic and Canton of Geneva, Switzerland (pp. 614–624). https://doi.org/10.1145/2736277.2741099 . · DOI
  26. Liao, Q. V., Fu, W. T., & Strohmaier, M. (2016). # Snowden: Understanding biases introduced by behavioral differences of opinion groups on social media. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3352–3363). ACM. · DOI
  27. Liu, Z., & Weber, I. (2014). Is twitter a public sphere for online conflicts? A cross-ideological and cross-hierarchical look. In International Conference on Social Informatics (pp. 336–347). Springer. · DOI
  28. Lu, H., Caverlee, J., Niu, W., & Biaswatch, A. (2015). A lightweight system for discovering and tracking topic-sensitive opinion bias in social media. In Proceedings of the 24th ACM international conference on information and knowledge management, CIKM ’15.
  29. Makazhanov, A., & Rafiei, D. (2013). Predicting political preference of twitter users. In Proceedings of advances in social networks analysis and mining, 289–305, ASONAM '13. · DOI
  30. Munson, S. A., Lee, S. Y., & Resnick, P. (2013c). Encouraging reading of diverse political viewpoints with a browser widget. In International AAAI conference on web and social media, ICWSM '13, AAAI.
  31. Munson, S. A., & Resnick, P. (2010). Presenting diverse political opinions: How and how much. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1457–1466). ACM. · DOI · Paper page
  32. Park, S., Kang, S., Chung, S., & Song, J. (2009). NewsCube: Delivering multiple aspects of news to mitigate media bias. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 443–452). ACM. · DOI · Paper page
  33. Pennacchiotti, M., & Popescu, A. M. (2011). Democrats, republicans and starbucks afficionados: User classification in twitter. In Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’11 (pp. 430–438). ACM, New York, NY, USA. https://doi.org/10.1145/2020408.2020477 . · DOI · Paper page
  34. Purver, M., & Karolina, S. (2015). Twitter language use reflects psychological differences between democrats and republicans. PLoS ONE, 10(9), e0137–e0422.
  35. Ribeiro, F. N., Lucas Henrique, F. B., Chakraborty, A., Kulshrestha, J., Babei, M., & Gummadi, K. P. (2015). Media bias monitor: Quantifying biases of social media news outlets at large-scale. In Proceedings of the 12th international AAAI conference of web and social media, ICWSM ’18.
  36. Semaan, B. C., Robertson, S. P., Douglas, S., & Maruyama, M. (2014). Social media supporting political deliberation across multiple public spheres: Towards depolarization. In Proceedings of the 17th ACM conference on computer supported cooperative work and social computing (pp. 1409–1421). ACM. · DOI
  37. Sharma, N. K., Ghosh, S., Benevenuto, F., Ganguly, N., & Gummadi, K. (2012). Inferring who-is-who in the twitter social network. ACM SIGCOMM Computer Communication Review, 42(4), 533–538. https://doi.org/10.1145/2377677.2377782 . · DOI
  38. Shi, Y., Mast, K., Weber, I., Kellum, A., & Macy, M. (2017). Cultural fault lines and political polarization. In Proceedings of the ACM conference on web science, WebSci ’17 (pp. 213–217). ACM, New York, NY, USA (2017). http://doi.acm.org/10.1145/3091478.3091520 . · DOI
  39. Smith, L. M., Zhu, L., Lerman, K., & Kozareva, Z. (2013). The role of social media in the discussion of controversial topics. In 2013 International conference on social computing (SocialCom), (pp. 236–243). IEEE. · DOI · Paper page
  40. Tavani, H. (2014). Search engines and ethics. In Zalta, E. N. (Ed.), The Stanford encyclopedia of philosophy, spring 2014 Edn.
  41. Teevan, J., Ramage, D., Morris, & M. R. (2011). #twittersearch: A comparison of microblog search and web search. In Proceedings of the 4th ACM international conference on web search and data mining, WSDM ’11 (pp. 35–44). ACM, New York, NY, USA. https://doi.org/10.1145/1935826.1935842 . · DOI · Paper page
  42. Van Couvering, E.(2010). Search engine bias: The structuration of traffic on the World-Wide Web. Ph.D. thesis, The London School of Economics and Political Science.
  43. Weber, I., Garimella, V. R. K., & Borra, E. (2012). Mining web query logs to analyze political issues. In Proceedings of the 4th annual ACM web science conference, WebSci ’12, pp. 330–334. ACM, New York, NY, USA. https://doi.org/10.1145/2380718.2380761 . · DOI
  44. Weber, I., Garimella, V. R. K., & Teka, A. (2013). Political hashtag trends. In European conference on information retrieval, ECIR '13 (pp. 857–860). Berlin: Springer.
  45. Welch, M. J., Cho, J., & Olston, C. (2011). Search result diversity for informational queries. In Proceedings of the 20th international conference on world wide web (pp. 237–246). ACM. · DOI
  46. Wong, F. M. F., Tan, C. W., Sen, S., & Chiang, M. (2016). Quantifying political leaning from tweets, retweets, and retweeters. IEEE Transactions on Knowledge and Data Engineering, 28, 2158. · DOI
  47. Yano, T., Resnik, P., & Smith, N. A. (2010). Shedding (a thousand points of) light on biased language. In Proceedings of NAACL HLT workshop on creating speech and language data with Amazon’s mechanical turk (CSLDAMT).
  48. Yilmaz, E., & Aslam, J. A. (2006). Estimating average precision with incomplete and imperfect judgments. In Proceedings of the 15th ACM international conference on information and knowledge management (pp. 102–111). ACM. · DOI · Paper page
  49. Yom-Tov, E., Dumais, S., & Guo, Q. (2013). Promoting civil discourse through search engine diversity. Social Science Computer Review, 32, 145–154. · DOI
  50. Zafar, M. B., Gummadi, K. P., & Danescu-Niculescu-Mizil, C. (2016). Message impartiality in social media discussions. In Proceedings in international AAAI conference on web and social media. ICWSM '16. AAAI.
  51. Zhou, D. X., Resnick, P., & Mei, Q. (2011). Classifying the political leaning of news articles and users from user votes. In Proceedings in international AAAI conference on web and social media. ICWSM '11. AAAI.

Cited by

See all 103

Metadata

Authors
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K.
Venue
Information Retrieval Journal
Published
2018-08-21
Volume
22
Issue
1-2
Pages
188-227
ISSN
  • 1386-4564 (ISSN-L)
  • 1573-7659
OpenAlex
W2888099522
Cited by
103
References
20
FWCI
16.87
Type
journal article
Language
en
Open access
hybrid
Licence
creativecommons.org/licenses/by/4.0 (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Search bias quantification: investigating political bias in social media and web search. https://app.cochat.ai/papers/10.1007/s10791-018-9341-2
The paper
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2019). Search bias quantification: investigating political bias in social media and web search. Information Retrieval Journal. https://doi.org/10.1007/s10791-018-9341-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.