Search bias quantification: investigating political bias in social media and web search
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Users frequently use search systems on the Web as well as online social media to learn about ongoing events and public opinion on personalities. Prior studies have shown that the top-ranked results returned by these search engines can shape user opinion about the topic (e.g., event or person) being searched. In case of polarizing topics like politics, where multiple competing perspectives exist, the political bias in the top search results can play a significant role in shaping public opinion towards (or away from) certain perspectives. Given the considerable impact that search bias can have on the user, we propose a generalizable search bias quantification framework that not only measures the political bias in ranked list output by the search system but also decouples the bias introduced by the different sources—input data and ranking system. We apply our framework to study the political bias in searches related to 2016 US Presidential primaries in Twitter social media search and find that both input data and ranking system matter in determining the final search output bias seen by the users. And finally, we use the framework to compare the relative bias for two popular search systems—Twitter social media search and Google web search—for queries related to politicians and political events. We end by discussing some potential solutions to signal the bias in the search results to make the users more aware of them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“This insight is particularly crucial in this digital era where many algorithms are trained using vast amounts of data (Barocas and Selbst 2014).”
BackgroundThe political blogosphere and the 2004 U.S. election: divided they blog
Adamic, L. A., & Glance, N. (2005) · Proceedings of the 3rd international workshop on Link discovery · doi:10.1145/1134271.1134277 · Cited by 2,619
Political Polarization on Twitter
Conover, M., Ratkiewicz, J., Francisco, M., Goncalves, B., Flammini, A., & Menczer, F. (2011) · Zenodo · doi:10.5281/zenodo.4589064 · Cited by 1,254
Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose
Morstatter, F., Pfeffer, J., Liu, H., & Carley, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v7i1.14401 · Cited by 863
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 3×
“Recently, the rise of algorithmic platforms’ influence on users’ online experience has motivated many studies (Datta et al. 2015; Sweeney 2013; Hannak et al. 2014; Sandvig et al. 2014; Executive Office of the President 2016) to audit these platforms and understand their biases.”
BackgroundMethodIn Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · cited 4×
“Prior field studies have shown that not only do the users place greater trust in highly ranked search results (Pan et al. 2007), but the opinions of undecided voters can be manipulated by biasing the search results about political candidates (Epstein and Robertson 2015).”
BackgroundInfluentialDynamic debates: an analysis of group polarization over time on Twitter
Yardi, S., & Boyd, D. (2010) · Bulletin of Science, Technology & Society · doi:10.1177/0270467610380011 · Cited by 625
Predicting the Political Alignment of Twitter Users
Conover, M. D., Goncalves, B., Ratkiewicz, J., Flammini, A., & Menczer, F. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.34 · Cited by 588
Research works printed but not linked: 51
- Babaei, M., Kulshrestha, J., Chakraborty, A., Benevenuto, F., Gummadi, K. P., & Weller, A. (2018). Purple feed: Identifying high consensus news posts on social media. In Proceedings of the AAAI/ACM conference on artifical intelligence, ethics & society, AIES 2018, New Orleans, USA. · DOI
- Baron, D. P. (2006). Persistent media bias. Journal of Public Economics, 90(1–2), 1–36. · DOI · Paper page
- Bhattacharya, P., Ghosh, S., Kulshrestha, J., Mondal, M., Zafar, M. B., Ganguly, N., & Gummadi, K. P. (2014a). Deep twitter diving: Exploring topical groups in microblogs at scale. In Proceedings of the 17th ACM conference on computer supported cooperative work and social computing, CSCW ’14, pp. 197–210. ACM, New York, NY, USA. http://doi.acm.org/10.1145/2531602.2531636 · DOI
- Bhattacharya, P., Zafar, M. B., Ganguly, N., Ghosh, S., & Gummadi, K. P. (2014b). Inferring user interests in the twitter social network. In Proceedings of the 8th ACM conference on recommender systems, RecSys ’14, pp. 357–360. ACM, New York, NY, USA. https://doi.org/10.1145/2645710.2645765 . · DOI · Paper page
- Bond, R., & Messing, S. (2015). Quantifying social media’s political space: Estimating ideology from publicly revealed preferences on facebook. American Political Science Review, 109(01), 62–78. · DOI · Paper page
- Carney, D. R., Jost, J. T., Gosling, S. D., & Potter, J. (2008). The secret lives of liberals and conservatives: Personality profiles, interaction styles, and the things they leave behind. Political Psychology, 29(6), 807–840. https://doi.org/10.1111/j.1467-9221.2008.00668.x. . · DOI · Paper page
- Chen, L., Mislove, A., & Wilson, C. (2015). Peeking beneath the hood of UBER. In Proceedings of the 2015 ACM conference on internet measurement conference (pp. 495–508). ACM. · DOI · Paper page
- Cohen, R., & Ruths, D. (2013). Classifying political orientation on twitter: It’s not easy!. In Proceedings of AAAI international conference on web & social media, ICWSM 2013, Boston, USA.
- Coletto, M., Garimella, K., Gionis, A., & Lucchese, C. (2017). A motif-based approach for identifying controversy. arXiv:1703.05053 . · arXiv
- Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. In Proceedings on privacy enhancing technologies, http://arxiv.org/abs/1408.6491 . · arXiv
- Epstein, R., & Robertson, R. E. (2015). The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of of the National Academy of Sciences (PNAS), 112(33), E4512–E4521. · DOI · Paper page
- Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017a). Suppressing the search engine manipulation effect (SEME). Proceedings ACM Human–Computer Interaction, 1, 42:1–42:22. https://doi.org/10.1145/3134677 . · DOI · Paper page
- Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017b). Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human–Computer Interaction, 1(2), 452.
- Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015). I always assumed that i wasn’t really that close to [her]: Reasoning about invisible algorithms in news feeds. In Proceedings of the 33rd annual ACM conference on human factors in computing systems (pp. 153–162). ACM. · DOI · Paper page
- things can be worse than they appear": Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of AAAI international conference on web & social media, ICWSM 2017 (pp. 62-71). · DOI · Paper page
- Fang, A., Ounis, I., Habel, P., Macdonald, C., & Limsopatham, N. (2015). Topic-centric classification of twitter user’s political orientation. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’15 (pp. 791–794).
- Fortunato, S., Flammini, A., Menczer, F., & Vespignani, A. (2006). Topical interests and the mitigation of search engine bias. Proceedings of of the National Academy of Sciences (PNAS), 103(34), 12684–12689. · DOI · Paper page
- Gentzkow, M., & Shapiro, J. (2010). What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1), 35–71. · DOI · Paper page
- Ghosh, S., Sharma, N., Benevenuto, F., Ganguly, N., & Gummadi, K. (2012). Cognos: Crowdsourcing search for topic experts in microblogs. In Proceedings of the 35th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’12 (pp. 575–590). ACM, New York, NY, USA. https://doi.org/10.1145/2348283.2348361 . · DOI · Paper page
- Golbeck, J., & Hansen, D. (2011). Computing political preference among Twitter followers. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1105–1108). ACM. · DOI
- Groseclose, T., & Milyo, J. (2005). A measure of media bias. The Quarterly Journal of Economics, 120(4), 1191–1237. · DOI · Paper page
- Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013). Measuring personalization of web search. In Proceedings of the 22nd international conference on world wide web, WWW ’13 (pp. 527–538). ACM, New York, NY, USA. https://doi.org/10.1145/2488388.2488435 . · DOI · Paper page
- Himelboim, I., McCreery, S., & Smith, M. (2013). Birds of a feather tweet together: Integrating network and content analyses to examine cross-ideology exposure on twitter. Journal of Computer-Mediated Communication, 18(2), 40–60. · DOI · Paper page
- Kliman-Silver, C., Hannak, A., Lazer, D., Wilson, C., & Mislove, A. (2015). Location, location, location: The impact of geolocation on web search personalization. In Proceedings of the 2015 internet measurement conference, IMC ’15 (pp. 121–127). ACM, New York, NY, USA. https://doi.org/10.1145/2815675.2815714 . · DOI
- Koutra, D., Bennett, P. N., & Horvitz, E. (2015). Events and controversies: Influences of a shocking news event on information seeking. In Proceedings of the 24th international conference on world wide web, WWW ’15, International world wide web conferences steering committee, Republic and Canton of Geneva, Switzerland (pp. 614–624). https://doi.org/10.1145/2736277.2741099 . · DOI
- Liao, Q. V., Fu, W. T., & Strohmaier, M. (2016). # Snowden: Understanding biases introduced by behavioral differences of opinion groups on social media. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3352–3363). ACM. · DOI
- Liu, Z., & Weber, I. (2014). Is twitter a public sphere for online conflicts? A cross-ideological and cross-hierarchical look. In International Conference on Social Informatics (pp. 336–347). Springer. · DOI
- Lu, H., Caverlee, J., Niu, W., & Biaswatch, A. (2015). A lightweight system for discovering and tracking topic-sensitive opinion bias in social media. In Proceedings of the 24th ACM international conference on information and knowledge management, CIKM ’15.
- Makazhanov, A., & Rafiei, D. (2013). Predicting political preference of twitter users. In Proceedings of advances in social networks analysis and mining, 289–305, ASONAM '13. · DOI
- Munson, S. A., Lee, S. Y., & Resnick, P. (2013c). Encouraging reading of diverse political viewpoints with a browser widget. In International AAAI conference on web and social media, ICWSM '13, AAAI.
- Munson, S. A., & Resnick, P. (2010). Presenting diverse political opinions: How and how much. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1457–1466). ACM. · DOI · Paper page
- Park, S., Kang, S., Chung, S., & Song, J. (2009). NewsCube: Delivering multiple aspects of news to mitigate media bias. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 443–452). ACM. · DOI · Paper page
- Pennacchiotti, M., & Popescu, A. M. (2011). Democrats, republicans and starbucks afficionados: User classification in twitter. In Proceedings of the 17th ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’11 (pp. 430–438). ACM, New York, NY, USA. https://doi.org/10.1145/2020408.2020477 . · DOI · Paper page
- Purver, M., & Karolina, S. (2015). Twitter language use reflects psychological differences between democrats and republicans. PLoS ONE, 10(9), e0137–e0422.
- Ribeiro, F. N., Lucas Henrique, F. B., Chakraborty, A., Kulshrestha, J., Babei, M., & Gummadi, K. P. (2015). Media bias monitor: Quantifying biases of social media news outlets at large-scale. In Proceedings of the 12th international AAAI conference of web and social media, ICWSM ’18.
- Semaan, B. C., Robertson, S. P., Douglas, S., & Maruyama, M. (2014). Social media supporting political deliberation across multiple public spheres: Towards depolarization. In Proceedings of the 17th ACM conference on computer supported cooperative work and social computing (pp. 1409–1421). ACM. · DOI
- Sharma, N. K., Ghosh, S., Benevenuto, F., Ganguly, N., & Gummadi, K. (2012). Inferring who-is-who in the twitter social network. ACM SIGCOMM Computer Communication Review, 42(4), 533–538. https://doi.org/10.1145/2377677.2377782 . · DOI
- Shi, Y., Mast, K., Weber, I., Kellum, A., & Macy, M. (2017). Cultural fault lines and political polarization. In Proceedings of the ACM conference on web science, WebSci ’17 (pp. 213–217). ACM, New York, NY, USA (2017). http://doi.acm.org/10.1145/3091478.3091520 . · DOI
- Smith, L. M., Zhu, L., Lerman, K., & Kozareva, Z. (2013). The role of social media in the discussion of controversial topics. In 2013 International conference on social computing (SocialCom), (pp. 236–243). IEEE. · DOI · Paper page
- Tavani, H. (2014). Search engines and ethics. In Zalta, E. N. (Ed.), The Stanford encyclopedia of philosophy, spring 2014 Edn.
- Teevan, J., Ramage, D., Morris, & M. R. (2011). #twittersearch: A comparison of microblog search and web search. In Proceedings of the 4th ACM international conference on web search and data mining, WSDM ’11 (pp. 35–44). ACM, New York, NY, USA. https://doi.org/10.1145/1935826.1935842 . · DOI · Paper page
- Van Couvering, E.(2010). Search engine bias: The structuration of traffic on the World-Wide Web. Ph.D. thesis, The London School of Economics and Political Science.
- Weber, I., Garimella, V. R. K., & Borra, E. (2012). Mining web query logs to analyze political issues. In Proceedings of the 4th annual ACM web science conference, WebSci ’12, pp. 330–334. ACM, New York, NY, USA. https://doi.org/10.1145/2380718.2380761 . · DOI
- Weber, I., Garimella, V. R. K., & Teka, A. (2013). Political hashtag trends. In European conference on information retrieval, ECIR '13 (pp. 857–860). Berlin: Springer.
- Welch, M. J., Cho, J., & Olston, C. (2011). Search result diversity for informational queries. In Proceedings of the 20th international conference on world wide web (pp. 237–246). ACM. · DOI
- Wong, F. M. F., Tan, C. W., Sen, S., & Chiang, M. (2016). Quantifying political leaning from tweets, retweets, and retweeters. IEEE Transactions on Knowledge and Data Engineering, 28, 2158. · DOI
- Yano, T., Resnik, P., & Smith, N. A. (2010). Shedding (a thousand points of) light on biased language. In Proceedings of NAACL HLT workshop on creating speech and language data with Amazon’s mechanical turk (CSLDAMT).
- Yilmaz, E., & Aslam, J. A. (2006). Estimating average precision with incomplete and imperfect judgments. In Proceedings of the 15th ACM international conference on information and knowledge management (pp. 102–111). ACM. · DOI · Paper page
- Yom-Tov, E., Dumais, S., & Guo, Q. (2013). Promoting civil discourse through search engine diversity. Social Science Computer Review, 32, 145–154. · DOI
- Zafar, M. B., Gummadi, K. P., & Danescu-Niculescu-Mizil, C. (2016). Message impartiality in social media discussions. In Proceedings in international AAAI conference on web and social media. ICWSM '16. AAAI.
- Zhou, D. X., Resnick, P., & Mei, Q. (2011). Classifying the political leaning of news articles and users from user votes. In Proceedings in international AAAI conference on web and social media. ICWSM '11. AAAI.
Cited by
A systematic review of worldwide causal and correlational evidence on digital media and democracy
Lorenz-Spreen, P., Oswald, L., Lewandowsky, S., & Hertwig, R. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01460-1 · Cited by 348
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Users choose to engage with more partisan news than they are exposed to on Google Search
Robertson, R. E., Green, J., Ruck, D. J., Ognyanova, K., Wilson, C., & Lazer, D. (2023) · Nature · doi:10.1038/s41586-023-06078-5 · Cited by 126
Auditing Algorithmic Bias on Twitter
Bartley, N., Abeliuk, A., Ferrara, E., & Lerman, K. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462491 · Cited by 62
Metadata
- Authors
- Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K.
- Venue
- Information Retrieval Journal
- Published
- 2018-08-21
- Volume
- 22
- Issue
- 1-2
- Pages
- 188-227
- ISSN
- 1386-4564 (ISSN-L)
- 1573-7659
- OpenAlex
- W2888099522
- Cited by
- 103
- References
- 20
- FWCI
- 16.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Search bias quantification: investigating political bias in social media and web search. https://app.cochat.ai/papers/10.1007/s10791-018-9341-2
- The paper
- Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2019). Search bias quantification: investigating political bias in social media and web search. Information Retrieval Journal. https://doi.org/10.1007/s10791-018-9341-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.