Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Search systems in online social media sites are frequently used to find information about ongoing events and people. For topics with multiple competing perspectives, such as political events or political candidates, bias in the top ranked results significantly shapes public opinion. However, bias does not emerge from an algorithm alone. It is important to distinguish between the bias that arises from the data that serves as the input to the ranking system and the bias that arises from the ranking system itself. In this paper, we propose a framework to quantify these distinct biases and apply this framework to politics-related queries on Twitter. We found that both the input data and the ranking system contribute significantly to produce varying amounts of bias in the search results and in different ways. We discuss the consequences of these biases and possible mechanisms to signal this bias in social media search systems' interfaces.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
The political blogosphere and the 2004 U.S. election: divided they blog
Adamic, L. A., & Glance, N. (2005) · Proceedings of the 3rd international workshop on Link discovery · doi:10.1145/1134271.1134277 · Cited by 2,619 · cited 1×
“While there have been prior attempts to infer the political bias of textual content in online forums, such as blogs and news articles [1, 46, 51] or hashtags on Twitter [44], to our knowledge, no work has measured the political bias of a tweet based on its textual content, particularly due to its very small size.”
BackgroundPolitical Polarization on Twitter
Conover, M., Ratkiewicz, J., Francisco, M., Goncalves, B., Flammini, A., & Menczer, F. (2011) · Zenodo · doi:10.5281/zenodo.4589064 · Cited by 1,254
Political Polarization on Twitter
Conover, M., Ratkiewicz, J., Francisco, M., et al. (2011) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4589065 · Cited by 1,005
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Self-citationDynamic debates: an analysis of group polarization over time on Twitter
Yardi, S., & Boyd, D. (2010) · Bulletin of Science, Technology & Society · doi:10.1177/0270467610380011 · Cited by 625
Predicting the Political Alignment of Twitter Users
Conover, M. D., Goncalves, B., Ratkiewicz, J., Flammini, A., & Menczer, F. (2011) · 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing · doi:10.1109/passat/socialcom.2011.34 · Cited by 588
The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections
Epstein, R., & Robertson, R. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1419828112 · Cited by 577
Research works printed but not linked: 4
- 10. USA Executive Office of the President. 2016. Big Data: A Report on Algorithmic Systems, Opportunity,and Civil Rights. http://tinyurl.com/Big-Data-White-House . (2016).
- 25. Amy Mitchell and Dana Page. 2015. The Evolving Role of News on Twitter and Facebook. Pew Research Center (2015).
- 34. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014).
- 39. Herman Tavani. 2014. Search Engines and Ethics. In The Stanford Encyclopedia of Philosophy (spring 2014 ed.), Edward N. Zalta (Ed.).
Cited by
The COVID-19 social media infodemic
Cinelli, M., Quattrociocchi, W., Galeazzi, A., Valensise, C. M., Brugnoli, E., Schmidt, A. L., Zola, P., Zollo, F., & Scala, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-73510-5 · Cited by 2,313
“f content and may amplify rumors and questionable information. Taking into account users’ preferences and attitudes, algorithms mediate and facilitate content promotion and thus information spreading [10]. This shift of paradigm profoundly impacts the construction of social perceptions [11] and the framing of narrativ…”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
“Given these risks, much algorithmic audit work has focused on goals to gauge user awareness of algorithmic bias [17, 26] or evaluate the impact of bias on user behaviour and outcomes [16, 19, 24], instead of directly challenging companies to change commercial systems.”
BackgroundFA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268
“Among most notable results, researchers have shown price discrimination in online retail sites [42], gender discrimination in job sites [16, 43], stereotypical gender roles re-enforced by online translation services [12] and image search [46], disparate performance on gender classi cation for Black women [13], and pol…”
BackgroundProblematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
“[94] 2017 Direct scrape Search Twitter Google”
Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval
Azzopardi, L. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446023 · Cited by 176
“Deciding on what party to support, whether to become vegan, or deciding to legalize cannabis, represent just a few examples of different socio-political decisions that people may turn to search engines to help inform their opinions [45, 54].”
BackgroundShen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
“Past research in this domain has uncovered harmful biases across a wide range of algorithmic systems such as search engines [57, 67, 74], housing websites [5], hiring systems [21], employment online services [15] and ecommerce platforms [47] .”
BackgroundSelf-citationMelki, J. P., Tamim, H., Hadid, D., Farhat, S., Makki, M. H., Ghandour, L., & Hitti, E. A. (2020) · Health Communication · doi:10.1080/10410236.2020.1858564 · Cited by 182
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155
Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Robertson, R. E., Lazer, D., & Wilson, C. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186143 · Cited by 139
“Utilizing this paradigm, researchers have conducted audits on algorithms in online marketplaces [12, 13, 33, 47, 48], search ranking bias on Twitter [38], user awareness of Facebook’s NewsFeed al-gorithm [24], localization of online maps [69], and, most relevant here, search engine rankings [32,…”
MethodAddressing age-related bias in sentiment analysis,
Diaz, M., Johnson, I., Lazar, A., Piper, A. M., & Gergle, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173986 · Cited by 138
“Other researchers have attempted to systematically sort out user bias (e.g., keywords that are put in) from system bias [43].”
BackgroundBuilding and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
“This includes examining broad classes of systems like search [14, 14, 25, 28, 32, 35, 36, 41, 53–55], e-commerce [10, 26], news recommendation [4, 31], online advertising [58, 60], maps [57], ridesharing [9, 30], online reviews and ratings [17, 18], natural language processing [7], and…”
BackgroundMetahuman systems = humans + machines that learn
Lyytinen, K., Nickerson, J. V., & King, J. L. (2021) · Journal of Information Technology · doi:10.1177/0268396220915917 · Cited by 122
“Organizations might not be ready now to cultivate their metahuman systems as suggested by multiple failures, examples of inherent bias, and systems running amok (Kulshrestha et al., 2017; Lambrecht and Tucker, 2019).”
BackgroundMining Disinformation and Fake News: Concepts, Methods, and Recent Advancements
Shu, K., Wang, S., Lee, D., & Liu, H. (2020) · Lecture notes in social networks · doi:10.1007/978-3-030-42699-6_1 · Cited by 185
Inside the Right-Leaning Echo Chambers: Characterizing Gab, an Unmoderated Social System
Lima, L., Reis, J. C. S., Melo, P., Murai, F., Araujo, L., Vikatos, P., & Benevenuto, F. (2018) · 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508809 · Cited by 107
Search bias quantification: investigating political bias in social media and web search
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2019) · Information Retrieval Journal · doi:10.1007/s10791-018-9341-2 · Cited by 103
“Our research contributions in this work can be summarized as follows: Finally, a version of the present work has been published earlier at a conference (Kulshrestha et al. 2017).”
MethodSelf-citationQuantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92
“ted (e.g., men are doctors, women are nurses), and that these results reinforced peoples’ perceptions of stereotypical gender roles [33]. Several groups have examined partisan slant in search results [13, 37, 45] in light of user studies demonstrating that partisan search results can significantly influence voting beh…”
BackgroundResultSuppressing the Search Engine Manipulation Effect (SEME)
Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134677 · Cited by 86
“It terms of Kulshrestha et al’s formulas [71], we found a ranking bias of -2.”
BackgroundMethodInfluentialFairness and Transparency in Ranking
Castillo, C. (2019) · ACM SIGIR Forum · doi:10.1145/3308774.3308783 · Cited by 118
To Live in Their Utopia: Why Algorithmic Systems Create Absurd Outcomes
Alkhatib, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445740 · Cited by 81
“…theoretical frameworks describing the social environments these systems produce and that designers create and foster, in the hope that some ontology or paradigm will motivate theoretically-grounded discussions about where the first two threads of research ought to lead [29–31, 36, 48, 70].”
BackgroundFacebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook
Silva, M., de Oliveira, L. S., Andreou, A., de Melo, P. O. S. V., Goga, O., & Benevenuto, F. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380109 · Cited by 103
“d sometimes similar but other times strikingly different discourses. Despite enabling public discourse, social media can leverage some problems such as biased content. For example, Kulshrestha et al. [29] proposed a framework to quantify bias in politics-related queriesonTwitter.Theyfoundthatboththeinputdataandtherank…”
BackgroundEcho Chambers, Rabbit Holes, and Algorithmic Bias: How YouTube Recommends Content to Real Users
Brown, M., Bisbee, J., Lai, A., Bonneau, R., Nagler, J., & Tucker, J. A. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4114905 · Cited by 71
“…in the study of political bias in what is recommended to users online, including in Google searches, Twitter searches, Twitter’s algorithmic timeline, and more (Robertson, Lazer and Wilson, 2018; Hannak et al., 2013; Kliman-Silver et al., 2015; Kulshrestha et al., 2017; Husz´ar et al., 2022).”
Rekabsaz, N., Kopeinik, S., & Schedl, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462949 · Cited by 55
Measuring Political Personalization of Google News Search
Le, H., Maragh, R., Ekdale, B., High, A., Havens, T., & Shafiq, Z. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313682 · Cited by 64
Practicing Information Sensibility: How Gen Z Engages with Online Information
Hassoun, A., Beacock, I., Consolvo, S., Goldberg, B., Kelley, P. G., & Russell, D. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581328 · Cited by 149
Auditing Algorithmic Bias on Twitter
Bartley, N., Abeliuk, A., Ferrara, E., & Lerman, K. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462491 · Cited by 62
“A complementary line of work focuses on disentangling the different sources of biases in search engines, specifically, controlling by the bias contained in the data that serves as the input to the algorithmic system [19].”
BackgroundPartisan search behavior and Google results in the 2018 U.S. midterm elections
Trielli, D., & Diakopoulos, N. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1764605 · Cited by 61
“Prior research has sought to determine the origin of bias, pointing to the impact of bias in the corpus of websites from which results are culled (Kulshrestha et al., 2017).”
BackgroundCruising Queer HCI on the DL: A Literature Review of LGBTQ+ People in HCI
Taylor, J., Simpson, E., Tran, A.-T., Brubaker, J. R., Fox, S. E., & Zhu, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642494 · Cited by 56
Mitigating Bias in Algorithmic Systems—A Fish-eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2023) · ACM Computing Surveys · doi:10.1145/3527152 · Cited by 52
“[119] propose an auditing technique where queries are submitted on Twitter, to measure bias on Twitter results as compared to search engines.”
BackgroundOn Measuring Bias in Online Information
Pitoura, E., Tsaparas, P., Flouris, G., Fundulaki, I., Papadakos, P., Abiteboul, S., & Weikum, G. (2018) · ACM SIGMOD Record · doi:10.1145/3186549.3186553 · Cited by 68
Scrutinizing assortative mating in birds
Wang, D., Forstmeier, W., Vâlcu, M., Dingemanse, N. J., Bulla, M., Both, C., Duckworth, R. A., Kiere, L. M., Karell, P., Albrecht, T., et al. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000156 · Cited by 54
Evaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices
Wolf, C., & Blomberg, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359245 · Cited by 48
“, justice and fairness [7, 43, 50, 82, 86]; bias [17, 22, 42, 46, 56, 72, 74]; or creepiness [47, 83]).”
BackgroundAuditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation
Juneja, P., & Mitra, T. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445250 · Cited by 72
“(iii) The ranking bias (rb) is introduced by the ranking algorithm of search engine [42].”
BackgroundMethodInfluentialInvestigating User Perception of Gender Bias in Image Search: The Role of Sexism
Otterbacher, J., Checco, A., Demartini, G., & Clough, P. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210094 · Cited by 45
Evaluation metrics for measuring bias in search engine results
Gezici, G., Lipani, A., Saygın, Y., & Yılmaz, E. (2021) · Information Retrieval · doi:10.1007/s10791-020-09386-w · Cited by 82
“For instance, in the course of elections, it is known that people issue repeated queries on the Web about political candidates and events such as “democratic debate”, “Donald Trump”, “climate change” (Kulshrestha et al. 2018).”
BackgroundMethodRaji, I. D., & Buolamwini, J. (2023) · Communications of the ACM · doi:10.1145/3571151 · Cited by 45
Joint Non-negative Matrix Factorization for Learning Ideological Leaning on Twitter
Lahoti, P., Garimella, K., & Gionis, A. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159669 · Cited by 58
“Unsupervised approaches have also been proposed, mainly based on the structure of user interests [23], social connections [4], and interactions [6, 16, 37].”
BackgroundKwakernaak, J., Eekhof, J. A., de Waal, M. W. M., Barenbrug, E. A. M., & Chavannes, N. H. (2019) · Journal of Medical Internet Research · doi:10.2196/12278 · Cited by 49
“There is evidence that obscuring the true identity of an information source, obscuring the affiliations of an information source, and control over user-generated content can greatly influence consumer health knowledge and behavior [31].”
Barlas, P., Kyriakou, K., Guest, O., Kleanthous, S., & Otterbacher, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432931 · Cited by 40
“[66] proposed an auditing technique that gauges information bias on Twitter, by benchmarking its behaviors against those of search engines.”
BackgroundBiases in the Facebook News Feed: A Case Study on the Italian Elections
Hargreaves, E., Agosti, C., Menasche, D., Neglia, G., Reiffers-Masson, A., & Altman, E. (2018) · 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508659 · Cited by 31
A Survey on Computational Politics
Haq, E.-U., Braud, T., Kwon, Y. D., & Hui, P. (2020) · IEEE Access · doi:10.1109/access.2020.3034983 · Cited by 50
Make Tweets Great Again: Who Are Opinion Leaders, and What Did They Tweet About Donald Trump?
Alexandre, I. A. R., Yoo, J., & Murthy, D. (2021) · Social Science Computer Review · doi:10.1177/08944393211008859 · Cited by 50
“Although tweets tend to be quite brief, personal or political biases can be detected in tweets (Kulshrestha et al., 2017).”
BackgroundWhite, man, and highly followed: gender and race inequalities in Twitter
Messias, J., Vikatos, P., & Benevenuto, F. (2017) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3106426.3106472 · Cited by 61
“There are several studies that retrieve information from social media to analyze and predict real-world phenomena such as stock market [3], migration [24], election [31, 40], and also political leaning of Twitter users [19, 33].”
BackgroundSelf-citationWhen Are Search Completion Suggestions Problematic?
Olteanu, A., Diaz, F., & Kazai, G. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415242 · Cited by 38
When Fair Ranking Meets Uncertain Inference
Ghosh, A., Dutt, R., & Wilson, C. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462850 · Cited by 58
Juneja, P., Bhuiyan, M. M., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580846 · Cited by 44
“hyper-personalization [4], ideological skew [6, 42, 71] ), discrimination (e.”
Kou, Y., Li, Y., Gui, X., & Suzuki-Gill, E. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174152 · Cited by 46
Purple Feed: Identifying High Consensus News Posts on Social Media
Babaei, M., Kulshrestha, J., Chakraborty, A., Benevenuto, F., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278761 · Cited by 34
“For instance, the political leaning of the publisher has been used to quantify the tweet’s leaning (Kulshrestha et al. 2017), or the leaning of news story URLs being shared by them on Face-book (Bakshy, Messing, and Adamic 2015).”
MethodSelf-citationHow Fair Can We Go: Detecting the Boundaries of Fairness Optimization in Information Retrieval
Gao, R., & Shah, C. S. (2019) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3341981.3344215 · Cited by 37
The case for voter-centered audits of search engines during political elections
Mustafaraj, E., Lurie, E., & Devine, C. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372835 · Cited by 46
“For example, the work by Kulshrestha et al. 2017 [23] introduced the concept of the interaction among various kinds of biases (input bias and ranking bias) in order to produce a more pronounced output bias.”
MethodDetecting Fake News on Social Media
Shu, K., & Liu, H. (2019) · Synthesis Lectures on Data Mining and Knowledge Discovery · doi:10.2200/s00926ed1v01y201906dmk018 · Cited by 83
Early Detection of Fake News with Multi-source Weak Social Supervision
Shu, K., Zheng, G., Li, Y., Mukherjee, S., Awadallah, A. H., Ruston, S. W. V., & Liu, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67664-3_39 · Cited by 41
“Specifically, we adopt the method in [10] to measure user bias (scores) by exploiting users’ interests over her historical tweets.”
MethodTurn to the Self in Human-Computer Interaction
Kou, Y., Gui, X., Chen, Y., & Nardi, B. A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300711 · Cited by 34
“There is increasing appreciation that people are experiencing various implications of technological advances, both individually and collectively, manifested in increasing societal and academic concerns such as algorithmic bias [60], privacy breaches [26], and online disinformation [93].”
BackgroundCrowdsourcing with Fairness, Diversity and Budget Constraints
Goel, N., & Faltings, B. V. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314282 · Cited by 36
“Other software systems have also been found to be biased against people of different races, genders and political views (Kay, Matuszek, and Munson 2015; Bolukbasi et al. 2016; Otter-bacher, Bates, and Clough 2017; Kulshrestha et al. 2017).”
BackgroundAddressing Bias and Fairness in Search Systems
Gao, R., & Shah, C. S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462807 · Cited by 37
“[19] developed a framework that quantified the amount of bias that arise from the data”
BackgroundLv, X., Chen, Y., & Guo, W. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.859597 · Cited by 31
RetractedCounteracting Bias and Increasing Fairness in Search and Recommender Systems
Gao, R., & Shah, C. S. (2020) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3383313.3411545 · Cited by 31
Featured Snippets and their Influence on Users’ Credibility Judgements
Bink, M., Zimmerman, S., & Elsweiler, D. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505766 · Cited by 31
“The way search queries are formulated influences the results returned [20].”
Auditing the representation of migrants in image web search results
Urman, A., Makhortykh, M., & Ulloa, R. (2022) · Humanities and Social Sciences Communications · doi:10.1057/s41599-022-01144-1 · Cited by 30
“Biases in web search outputs in fl uence public opinion and perceptions of social reality (Epstein and Robertson, 2015; Kulshrestha et al., 2017; Allam et al., 2014).”
BackgroundViewpoint Diversity in Search Results
Draws, T., Roy, N., Inel, O., Rieger, A., Hada, R., Yalcin, M. O., Timmermans, B., & Tintarev, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_18 · Cited by 29
Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images
Xu, S., Hou, D., Pang, L., Deng, J., Xu, J., Shen, H., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657750 · Cited by 45
“Kulshrestha et al. [22] formulated a framework for quantifying bias, addressing its origins in both the data source and the ranking system.”
BackgroundChen, N., Liu, J., Dong, X., Liu, Q., Sakai, T., & Wu, X.-M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698420 · Cited by 51
Auditing the Information Quality of News-Related Queries on the Alexa Voice Assistant
Dambanemuya, H. K., & Diakopoulos, N. A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449157 · Cited by 28
“This includes the auditing of informational biases that may result from the data or algorithms driving these systems [32], which may in turn result in increased or decreased exposure to different (1) sources of information, and (2) types of content.”
BackgroundRepresentativeness and face-ism: Gender bias in image search
Ulloa, R., Richter, A. C., Makhortykh, M., Urman, A., & Kacperski, C. (2022) · New Media & Society · doi:10.1177/14614448221100699 · Cited by 31
“In the future, the validity of findings could be improved with a rank-based approach (e.g. Kulshrestha et al., 2017); this was not a viable option for the current study as image search outputs are presented in a grid that varies according to screen sizes and involves less-structured visual…”
BackgroundDoes algorithmic filtering lead to filter bubbles in online tourist information searches?
Gong, Y., Schroeder, A., Pan, B., Sundar, S. S., & Mowen, A. J. (2023) · Information Technology & Tourism · doi:10.1007/s40558-023-00279-4 · Cited by 29
“Previous YouTube algorithm audit studies have found evidence of filter bubbles in political topics (e.g., O’Callaghan et al. 2015; Kulshrestha et al. 2017; Le et al. 2019).”
Ulloa, R., & Kacperski, C. (2023) · New Media & Society · doi:10.1177/14614448231154926 · Cited by 22
“The ranking effect is so widely accepted that heavy weights are used in studies that investigate biases in search engines (Kulshrestha et al., 2017; Robertson et al., 2018).”
BackgroundEslami, M. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3022198.3024947 · Cited by 17
“We have discussed the consequence of such biases and how to build a bias-aware design and increase users’ awareness of these potential biases [4].”
BackgroundSelf-citationPanda, A., Mothilal, R. K., Choudhury, M., Bali, K., & Pal, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392860 · Cited by 25
Show me a "Male Nurse"! How Gender Bias is Reflected in the Query Formulation of Search Engine Users
Kopeinik, S., Mara, M., Ratz, L., Krieg, K., Schedl, M., & Rekabsaz, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580863 · Cited by 25
The association of social media with dietary behaviors among adults in the United Arab Emirates
Ismail, L. C., Osaili, T. M. I., Naja, F., Wartanian, M., Elkabat, G., Arnous, M., Alkoukou, H., Mohamad, M. N., Saleh, S. T., Daour, R. A., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35574 · Cited by 20
“This way, SMP provides personalized content to users and enhances their experience [6].”
BackgroundOnline Post-Processing in Rankings for Fair Utility Maximization
Gupta, A., Johnson, E., Payan, J., Roy, A. K., Kobren, A., Panda, S., Tristan, J.-B., & Wick, M. (2021) · Proceedings of the 14th ACM International Conference on Web Search and Data Mining · doi:10.1145/3437963.3441724 · Cited by 16
“investigate and distinguish between sources of bias arising from the data and the ranking model respectively [16].”
BackgroundElection Fraud, YouTube, and Public Perception of the Legitimacy of President Biden
Bisbee, J. H., Brown, M. A., Lai, A., Bonneau, R., Nagler, J., & Tucker, J. A. (2022) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i3.60 · Cited by 17
“…auditingmethodology to online contexts to identify political bias in Google search results, Twitter search results, Twitter’s news feed ranking system, and more (Robertson, Lazer, and Wilson 2018; Hannak et al. 2013; Kliman-Silver et al. 2015; Kulshrestha et al. 2017; Huszár et al. 2022).”
BackgroundAlgorithmic Audit of Italian Car Insurance: Evidence of Unfairness in Access and Pricing
Fabris, A., Mishler, A., Gottardi, S., Carletti, M., Daicampi, M., Susto, G. A., & Silvello, G. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462569 · Cited by 21
“2014), racial bias in judicial risk assessment (Skeem and Lowenkamp 2016), sources of bias in political queries on Twitter (Kulshrestha et al. 2017), genderand race-based differences in accuracy of face analysis technology (Buolamwini and Gebru 2018) and radicalization on YouTube (Ribeiro et al.”
Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T.-H., & Wilson, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604667 · Cited by 42
Gender Fairness in Information Retrieval Systems
Bigdeli, A., Arabzadeh, N., Seyedsalehi, S., Zihayat, M., & Bagheri, E. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532680 · Cited by 22
“[21] examine the impact of such biases in the context of political search queries.”
BackgroundSchoultz, M., Leung, J., Bonsaksen, T., Ruffolo, M. C., Thygesen, H., Price, D., & Geirdal, A. Ø. (2021) · Healthcare · doi:10.3390/healthcare9060735 · Cited by 19
“This is because social media algorithms account for people’s preferences and facilitate content promotion, and therefore, the spread of information [17].”
Characterizing Scalability Issues in Spreadsheet Software using Online Forums
Mack, K., Lee, J., Chang, K. C., Karahalios, K. G., & Parameswaran, A. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3174359 · Cited by 14
“Adapting the methodology used in [19, 21, 18], we finalized the set of search terms in the following manner: (1) From the randomly collected posts, we identified scalability posts and their keywords.”
BackgroundMethodSelf-citationWeaponising Social Media for Information Divide and Warfare
Haq, E.-U., Gareth Tyson, A., Braud, T., & Hui, P. (2022) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3511095.3536372 · Cited by 21
Wang, B., & Liu, J. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672520 · Cited by 18
“Firstly, users might be exposed to biases in search results with algorithmic biases originating from the training data and retrieval algorithms (e.g., presenting results with ranking or diversity bias and information with unfairness and inequity) [22, 34, 62].”
Lee, S., Cockburn, L., & Nganji, J. T. (2022) · Media War & Conflict · doi:10.1177/17506352221103487 · Cited by 12
Fleischmann, K. R., Greenberg, S. R., Gurari, D., Stangl, A. J., Verma, N., Day, J. R., Simons, R. N., & YEH, T. P. (2019) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3311957.3359437 · Cited by 11
“In terms of bias mitigation, research in CSCW has explored a range of forms of bias, including ethnic bias [2], gender bias [21,25], geographical bias [20], and political bias [15].”
BackgroundFeng, Y., Gu, W., Dong, F., Dong, D., & Qiao, Z. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02048-z · Cited by 13
A Versatile Framework for Evaluating Ranked Lists in Terms of Group Fairness and Relevance
Sakai, T., Kim, J. Y., & Kang, I.-H. (2023) · ACM Transactions on Information Systems · doi:10.1145/3589763 · Cited by 24
“5 Polarity on a Binary Atribute Set Consider a situation with a single binary attribute set, A = {a1,a2} [38, 49].”
BackgroundMethodComplex Mediation in the Formation of Political Opinions
Kou, Y., & Nardi, B. A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174210 · Cited by 11
“Yardi and Boyd studied Twitter discussions about the shooting of a doctor who performed late-term abortions [110], and found that people were limited in their ability to engage in meaningful discussion with others with different viewpoints.”
BackgroundInfluentialUnderstanding Motivations behind Inaccurate Check-ins
Xu, F., Zhang, G., Chen, Z., Huang, J., Li, Y., Yang, D., Zhao, B. Y., & Meng, F. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274457 · Cited by 14
Šimko, J., Tomlein, M., Pecher, B., Móro, R., Srba, I., Štefancová, E., Hrčková, A., Kompan, M., Podroužek, J., & Bieliková, M. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450614.3463353 · Cited by 11
“, to distinguish and quantify sources of bias in search results [13].”
BackgroundPark, J., Ellezhuthil, R. D., Isaac, J., Mergerson, C., Feldman, L., & Singh, V. K. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583617 · Cited by 14
“Similarly, multiple recent studies have reported on political bias in the algorithms used by YouTube recommendation [37] and Twitter search [33].”
BackgroundChanging landscape of fake news research on social media: a bibliometric analysis
Sahid, A., Maleh, Y., & Ouazzane, K. (2025) · Quality & Quantity · doi:10.1007/s11135-024-02048-9 · Cited by 11
Errors in Geotargeted Display Advertising: Good News for Local Journalism?
Bandy, J., & Hecht, B. J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449166 · Cited by 13
“[60] explored how social media platforms affect news sharing practices, and some algorithm audits [53,70] have explored how search engine results treat news content.”
BackgroundList-wise Fairness Criterion for Point Processes
Shang, J., Sun, M., & Lam, N. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403246 · Cited by 10
On the Characteristics of Ranking-based Gender Bias Measures
Klasnja, A., Arabzadeh, N., Mehrvarz, M., & Bagheri, E. (2022) · ACM Web Science Conference (WebSci) · doi:10.1145/3501247.3531540 · Cited by 10
“Later, inspired by [7, 11, 23], a new gender bias metric was proposed in which the ranking properties of the retrieved list of documents were more taken into consideration.”
BackgroundWhose Walkability? Challenges in Algorithmically Measuring Subjective Experience
Díaz, M., & Diakopoulos, N. A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359228 · Cited by 9
“Recent research has begun to focus on how to make data-driven systems transparent and intelligible to end users [17, 37, 50], underlining a need to help users understand algorithmic outputs.”
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 230
Ismail, L. C., Osaili, T. M. I., Obaid, R. S., Hashim, M., Ahmed, M. K., Al-Fayadh, F., Farah, A., Sad, H., Alghanem, H., Mohamad, M. N., et al. (2025) · BMC Public Health · doi:10.1186/s12889-024-20836-8 · Cited by 9
Khoo, L. S., Bay, J. Q., Yap, M. L. K., Lim, M. K., Chong, C. Y., Yang, Z., & Lo, D. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00066 · Cited by 14
“[34] mentioned that individuals’ personal biases might be explicit in their writings.”
BackgroundAzzopardi, L. A., & Liu, J. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638297 · Cited by 10
Leveraging the Facebook ads platform for election polling
Ribeiro, F. N., Kansaon, D. P., & Benevenuto, F. (2019) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3323503.3349552 · Cited by 40
“Os dados coletados nessas redes on-line também foram examinados no contexto eleitoral para predizer eleições [2, 27], pesquisas eleitorais [28] e para detectar o viés ide-ológico de usuários [11, 26].”
BackgroundNewsGuesser: Using Curiosity to Reduce Selective Exposure
Sun, L., Zhang, H., Liu, E., Liu, M., & Vaccaro, K. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637376 · Cited by 7
Low, N. K. Y., & Melatos, A. (2021) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2021.126722 · Cited by 8
“Substantial resources have been invested recently to refine the theoretical definition of media bias and devise algorithms to detect it in practice [3, 4, 5, 6, 7] (see [1] for a review).”
Background
Metadata
- Authors
- Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K.
- Affiliations
- Max Planck Institute for Software Systems (DE)
- University of Illinois Urbana-Champaign (US)
- Indian Institute of Engineering Science and Technology, Shibpur (IN)
- ORCID
- Venue
- Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
- Publisher
- ACM
- Published
- 2017-02-25
- Pages
- 417-432
- OpenAlex
- W4300539796
- Cited by
- 148
- References
- 45
- Citations by year
- 2026: 7 · 2025: 9 · 2024: 15 · 2023: 18 · 2022: 21 · 2021: 28 · 2020: 16 · 2019: 15 · 2018: 15 · 2017: 4
- FWCI
- 10.50
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Computer Supported Cooperative Work (CSCW) (publishedVersion)
- Funders
- NSF (CHS-1564041)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- search bias, social media search, ranking bias, data bias, Twitter, search result ranking, public opinion, bias quantification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.