Auditing Partisan Audience Bias within Google Search
The DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias and personalization within Google Search. Following Donald Trump's inauguration, we recruited 187 participants to complete a survey and install a browser extension that enabled us to collect Search Engine Results Pages (SERPs) from their computers. To quantify partisan audience bias, we developed a domain-level score by leveraging the sharing propensities of registered voters on a large Twitter panel. We found little evidence for the "filter bubble'' hypothesis. Instead, we found that results positioned toward the bottom of Google SERPs were more left-leaning than results positioned toward the top, and that the direction and magnitude of overall lean varied by search query, component type (e.g. "answer boxes"), and other factors. Utilizing rank-weighted metrics that we adapted from prior work, we also found that Google's rankings shifted the average lean of SERPs to the right of their unweighted average.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research on AI auditing has focused on evaluating whether specific applications meet predefined industry requirements.”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Robertson et al. 2018
In the citing paper
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Robertson et al. 2018
In the citing paper
“For example, it is well-known that certain users on advertising platforms are more valuable to advertisers than others [48,55,65].”
From Discrimination through Optimization (2019) · cites this paper as [65]
In the citing paper
“Some researchers have leveraged narrow conceptions of auditing to test for bias and discrimination in online ad delivery (Ali et al., 2019;Sweeney, 2013) and autocomplete algorithms (Robertson et al., 2018), for fairness in image classification systems (Morina et al., 2019), for accuracy in news curation systems (Bandy & Diakopoulos, 2019), for completeness in datasets (Coston et al., 2021;Sookhak”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Robertson et al., 2018
Citations
Cites
Birds of a Feather: Homophily in Social Networks
McPherson, M., Smith-Lovin, L., & Cook, J. M. (2001) · Annual Review of Sociology · doi:10.1146/annurev.soc.27.1.415 · Cited by 18,567
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“Spurred by these concerns, research on the spread of fake news on social media has flourished [1, 51, 60, 96].”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 15×
“The resulting bias score is continuous and scales from -1 to 1, where a score of -1 means that a domain i was shared exclusively by Democrats (ri = 0, di > 0), and a score of 1 means that a domain i was shared exclusively by Republicans (ri > 0, di = 0).”
BackgroundMethodResultInfluential(2011) · Cited by 4,430
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Self-citationProlific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationExposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 1×
“We utilized two established crowdsourcing services [72, 75] to recruit a demographically diverse sample of 187 participants to take our survey.”
Method
Research works printed but not linked: 17
- Nicholas J. Belkin. 1980. Anamolous States of Knowledge as a Basis for Information Retrieval. Canadian Journal of Information and Library Sciences 5 (1980), 133--143.
- Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
- William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: A Cross-National Survey. (2017).
- Google. 2016. Google Trends: Year in Search. Google Trends. (2016). https://trends.google.com/trends/yis/2016/ GLOBAL
- Google. 2017. Google Chrome Privacy Notice. (2017). https://www.google.com/chrome/browser/privacy/ #browser-modes Accessed: 2017-04-01. {42} Jeffrey Gottfried and Elisa Shearer. 2016. News use across social media platforms. Pew Research Center. (2016). http://www.journalism.org/2016/05/26/news-use-across-social-media-platforms-2016/
- Cecilia Kang, Tiffany Hsu, Kevin Roose, Natasha Singer, and Matthew Rosenberg. 2018. Mark Zuckerberg Testimony: Day 2 Brings Tougher Questioning. The New York Times (April 2018).
- Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.
- Amy Mitchell, Jeffrey Gottfried, Jocelyn Kiley, and Katerina Eva Matsa. 2014. Political Polarization & Media Habits. Pew Research Center's Journalism Project. (Oct. 2014).
- Nic Newman, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. SSRN Electronic Journal (2017).
- 10.5555/2029079 · doi:10.5555/2029079
- Media Insight Project. 2014. The Personal News Cycle: How Americans Choose to Get Their News. Media Insight Project. (March 2014). https://www.americanpressinstitute.org/publications/reports/survey-research/ how-americans-get-news/
- Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of "Data and Discrimination: Converting Critical Concerns into Productive Inquiry", a Productivereconference at the 64th Annual Meeting of the International Communication Association.
- Jennifer A Stark and Nicholas Diakopoulos. 2017. Using Baselines for Algorithm Audits. In European Conference on Data and Computational Journalism. 3.
- Danny Sullivan. 2016. Google Now Handles at Least 2 Trillion Searches per Year. Search Engine Land. (May 2016). https://searchengineland.com/google-now-handles-2--999-trillion-searches-per-year-250247
- New York Times. 1995. Edward Bernays, 'Father of Public Relations' And Leader in Opinion Making, Dies at 103. (1995).
- Francesca Tripodi. 2018. Searching for Alternative Facts: Analyzing Scriptural Inference in Conservative News Practices. Data & Society. (May 2018).
Cited by
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Discrimination through Optimization
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359301 · Cited by 268
“eotypical gender roles re-enforced by online translation services [12] and image search [43], disparate performance on gender classi˙cation for Black women [13], and political partisanships in search [19, 47, 58]. Although most of the work focused exclusively on the algorithms themselves, recently researchers began to…”
BackgroundMeasuring Misinformation in Video Search Platforms: An Audit Study on YouTube
Hussein, E., Juneja, P., & Mitra, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392854 · Cited by 226
Laine, J., Minkkinen, M., & Mäntymäki, M. (2024) · Information & Management · doi:10.1016/j.im.2024.103969 · Cited by 227
Ohme, J., Araujo, T., Boeschoten, L., Freelon, D., Ram, N., Reeves, B. B., & Robinson, T. N. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2023.2181319 · Cited by 208
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024) · Review of Accounting Studies · doi:10.1007/s11142-024-09833-9 · Cited by 195
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Propaganda and Misinformation on Facebook and Twitter during the Russian Invasion of Ukraine
Pierri, F., Luceri, L., Jindal, N., & Ferrara, E. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583597 · Cited by 180
Search as News Curator: The Role of Google in Shaping Attention to News Information
Trielli, D., & Diakopoulos, N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300683 · Cited by 107
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
“used a survey and web browser extension to collect users’ experiences, in order to audit partisan audience biases in Google search results [74].”
BackgroundMethodExamining the consumption of radical content on YouTube
Hosseinmardi, H., Ghasemian, A., Clauset, A., Mobius, M., Rothschild, D. M., & Watts, D. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101967118 · Cited by 171
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152
“…conceptions of auditing to test for bias and discrimination in online ad delivery (Ali et al., 2019; Sweeney, 2013) and autocomplete algorithms (Robertson et al., 2018), for fairness in image classification systems (Morina et al., 2019), for accuracy in news curation systems (Bandy &…”
BackgroundAuditing Algorithms: Understanding Algorithmic Systems from the Outside In
Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000083 · Cited by 140
Self-citationNeutral bots probe political bias on social media
Chen, W., Pacheco, D., Yang, K.-C., & Menczer, F. (2021) · Nature Communications · doi:10.1038/s41467-021-25738-6 · Cited by 64
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
Self-citationLam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 77
Users choose to engage with more partisan news than they are exposed to on Google Search
Robertson, R. E., Green, J., Ruck, D. J., Ognyanova, K., Wilson, C., & Lazer, D. (2023) · Nature · doi:10.1038/s41586-023-06078-5 · Cited by 126
Self-citationDiscrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased OutcomesPreprint
Ali, M., Sapieżyński, P., Bogen, M., Korolova, A., Mislove, A. E., & Rieke, A. (2019) · arXiv (Cornell University) · Cited by 126
Jiang, S., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274351 · Cited by 125
“For example, in-lab experiments have shown that exposure to biased information online [102] may significantly impact voting behavior [22, 23], while naïve information sharing may promote homophilous “echo chambers” of information [21, 57, 58].”
BackgroundSelf-citationNo Polarization From Partisan News: Over-Time Evidence From Trace Data
Wojcieszak, M., de Leeuw, S., Menchen-Trevino, E., Lee, S., Huang-Isherwood, K. M., & Weeks, B. (2023) · The International Journal of Press/Politics · doi:10.1177/19401612211047194 · Cited by 97
Tuning EU equality law to algorithmic discrimination: Three pathways to resilience
Xenidis, R. (2020) · Maastricht Journal of European and Comparative Law · doi:10.1177/1023263x20982173 · Cited by 123
Quantifying partisan news diets in Web and TV audiences
Muise, D., Hosseinmardi, H., Howland, B., Mobius, M., Rothschild, D., & Watts, D. J. (2022) · Science Advances · doi:10.1126/sciadv.abn0083 · Cited by 72
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2023) · arXiv · doi:10.48550/arxiv.2304.00228 · Cited by 115
Lewandowsky, S., Smillie, L., García, D., Hertwig, R., Weatherall, J., Egidy, S., Robertson, R. E., O’Connor, C., Kozyreva, A., Lorenz-Spreen, P., et al. (2020) · Joint Research Centre (European Commission) · doi:10.2760/709177 · Cited by 113
Self-citationCountering Algorithmic Bias and Disinformation and Effectively Harnessing the Power of AI in Media
Shin, D., Hameleers, M., Park, Y. J., Kim, J. N., Trielli, D., Diakopoulos, N. A., Helberger, N., Lewis, S. C., Westlund, O., & Baumann, S. (2022) · Journalism & Mass Communication Quarterly · doi:10.1177/10776990221129245 · Cited by 111
Rafieian, O., & Yoganarasimhan, H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1548-643520230000020004 · Cited by 110
Web scraping for research: Legal, ethical, institutional, and scientific considerations
Brown, M. A., Gruen, A., Maldoff, G., Messing, S., Sanderson, Z., & Zimmer, M. T. (2025) · Big Data & Society · doi:10.1177/20539517251381686 · Cited by 108
Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News
Bandy, J., & Diakopoulos, N. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7277 · Cited by 64
DeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98
Balayn, A., Lofi, C., & Houben, G.-J. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00671-8 · Cited by 98
“[44] address age bias in sentiment analysis—social media news and existing polarization biases [45], fairness in self-driving vehicles [75], text processing [104]), web information systems and biases arising from them [43,110,121,125,135,144,149,163,171].”
BackgroundMökander, J., & Axente, M. (2023) · AI & SOCIETY · doi:10.1007/s00146-021-01286-x · Cited by 62
Can Unconfident LLM Annotations Be Used for Confident Conclusions?
Gligorić, K., Zrnic, T., Lee, C., Candès, E. J., & Jurafsky, D. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.179 · Cited by 94
Measuring Political Personalization of Google News Search
Le, H., Maragh, R., Ekdale, B., High, A., Havens, T., & Shafiq, Z. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313682 · Cited by 64
Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92
“Thus, although eye-tracking studies are available for sites like Google Search [11, 20, 21, 44], the data may not be applicable to other search engines.”
BackgroundMethodInfluentialSelf-citationWojcieszak, M., Menchen-Trevino, E., Goncalves, J. F. F., & Weeks, B. (2022) · The International Journal of Press/Politics · doi:10.1177/19401612211009160 · Cited by 72
Diminished diversity-of-thought in a standard large language model
Park, P. S., Schoenegger, P., & Zhu, C. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02307-x · Cited by 83
“…and attitudes, the tendency of Google users’ Search Engine Results Pages (SERPs) to be more right-leaning near the top than near the bottom (Robertson, 2018), the finding that the 40 most prominent websites in right-learning media have 2.7 times the total visibility on Google Search when…”
BackgroundInfluentialEvaluation metrics for measuring bias in search engine results
Gezici, G., Lipani, A., Saygın, Y., & Yılmaz, E. (2021) · Information Retrieval · doi:10.1007/s10791-020-09386-w · Cited by 82
#TulsaFlop: A Case Study of Algorithmically-Influenced Collective Action on TikTokPreprint
Bandy, J., & Diakopoulos, N. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07716 · Cited by 82
Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
Yang, K.-C., & Menczer, F. (2025) · Proceedings of the 17th ACM Web Science Conference 2025 · doi:10.1145/3717867.3717903 · Cited by 81
“As prominent platforms for information-seeking, search engines have been scrutinized by many researchers evaluating if their ranking algorithms favor certain news sources (Robertson et al. 2018; Trielli and Diakopoulos 2019).”
Haroon, M., Chhabra, A., Liu, X., Mohapatra, P., Shafiq, Z., & Wojcieszak, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.10666 · Cited by 79
Search Media and Elections: A Longitudinal Investigation of Political Search Results
Metaxa, D., Park, J. S., Landay, J. A., & Hancock, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359231 · Cited by 39
Misinformation, Disinformation, and Generative AI: Implications for Perception and Policy
Jaidka, K., Chen, T., Chesterman, S. A., Hsu, W., Kan, M., Kankanhalli, M., Lee, M. L., Seres, G., Sim, T., Taeihagh, A., et al. (2024) · Digital Government Research and Practice · doi:10.1145/3689372 · Cited by 75
Urman, A., Makhortykh, M., & Ulloa, R. (2021) · Social Science Computer Review · doi:10.1177/08944393211006863 · Cited by 75
Partisan search behavior and Google results in the 2018 U.S. midterm elections
Trielli, D., & Diakopoulos, N. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1764605 · Cited by 61
The political effects of X’s feed algorithm
Gauthier, G., Hodler, R., Widmer, P., & Zhuravskaya, E. V. (2026) · Nature · doi:10.1038/s41586-026-10098-2 · Cited by 73
Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation
Juneja, P., & Mitra, T. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445250 · Cited by 72
Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019) · arXiv · doi:10.48550/arxiv.1904.02095 · Cited by 72
Auditing local news presence on Google News
Fischer, S., Jaidka, K., & Lelkes, Y. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-00954-0 · Cited by 71
Mitigating Bias in Algorithmic Systems—A Fish-eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2023) · ACM Computing Surveys · doi:10.1145/3527152 · Cited by 52
Auditing the Partisanship of Google Search Snippets
Hu, D., Jiang, S., E. Robertson, R., & Wilson, C. (2019) · The World Wide Web Conference · doi:10.1145/3308558.3313654 · Cited by 51
Self-citationSharing without clicking on news in social media
Sundar, S. S., Snyder, E. C., Liao, M., Yin, J., Wang, J., & Chi, G. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02067-4 · Cited by 72
Nudging recommendation algorithms increases news consumption and diversity on YouTube
Yu, X., Haroon, M., Menchen-Trevino, E., & Wojcieszak, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae518 · Cited by 72
Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
Lam, M. S., Pandit, A., Kalicki, C. H., Gupta, R., Sahoo, P., & Metaxa, D. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610209 · Cited by 39
Steiner, M., Magin, M., Stark, B., & Geiß, S. (2022) · Information, Communication & Society · doi:10.1080/1369118x.2020.1776367 · Cited by 48
Dash, A., Chakraborty, A., Ghosh, S., Mukherjee, A., & Gummadi, K. P. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445944 · Cited by 66
Curating Quality? How Twitter’s Timeline Algorithm Treats Different Types of News
Bandy, J., & Diakopoulos, N. (2021) · Social Media + Society · doi:10.1177/20563051211041648 · Cited by 40
Auditing Algorithmic Bias on Twitter
Bartley, N., Abeliuk, A., Ferrara, E., & Lerman, K. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462491 · Cited by 62
“For example, the introduction of “who to follow” friend recommendation on Twitter, disproportionately accelerated the growth of the already-popular accounts [31]; Google search results of political queries differ significantly based on users’ previous browsing history [26].”
BackgroundDo, K., Pang, R. Y., Jiang, J., & Reinecke, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581347 · Cited by 62
Choudhary, T. (2024) · IEEE Access · doi:10.1109/access.2024.3523764 · Cited by 62
Bias Misperceived:The Role of Partisanship and Misinformation in YouTube Comment Moderation
Jiang, S., Robertson, R. E., & Wilson, C. (2019) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v13i01.3229 · Cited by 61
Self-citationNechushtai, E., Zamith, R., & Lewis, S. C. (2023) · Mass Communication & Society · doi:10.1080/15205436.2023.2173609 · Cited by 60
When Fair Ranking Meets Uncertain Inference
Ghosh, A., Dutt, R., & Wilson, C. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462850 · Cited by 58
Self-citationAn external stability audit framework to test the validity of personality prediction in AI hiring
Rhea, A. K., Markey, K., D’Arinzo, L., Schellmann, H., Sloane, M., Squires, P., Khan, F. A., & Stoyanovich, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00861-0 · Cited by 57
Algorithms that "Don't See Color": Measuring Biases in Lookalike and Special Ad Audiences
Sapieżyński, P., Ghosh, A., Kaplan, L., Rieke, A., & Mislove, A. E. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534135 · Cited by 58
Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications
Morales‐Navarro, L., Kafai, Y. B., Konda, V., & Metaxa, D. (2024) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3628516.3655752 · Cited by 55
How search engines disseminate information about COVID-19 and why they should do better
Makhortykh, M., Urman, A., & Ulloa, R. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-017 · Cited by 54
“To account for differences in search result ranking, we used RBO, a metric designed explicitly for investigating similarities in the output of search engines (Webber et al., 2010; Cardoso & Magalhães, 2011; Robertson et al., 2018).”
MethodMore Accounts, Fewer Links: How Algorithmic Curation Impacts Media Exposure in Twitter Timelines
Bandy, J., & Diakopoulos, N. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449152 · Cited by 54
“The substantial dependence on mainstream media outlets is evidenced in several studies of Google News [38, 71], Google search [20, 52, 77, 92], and Apple News [9], suggesting that algorithmic curation may exacerbate existing disparities in the media industry and exclude content from local and…”
BackgroundMethodResultInfluentialPolitical Polarization in Online News Consumption
Garimella, K., Smith, T., Weiss, R., & West, R. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18049 · Cited by 54
Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies
Vecchione, B., Levy, K., & Barocas, S. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483294 · Cited by 51
Scaling up search engine audits: Practical insights for algorithm auditing
Ulloa, R., Makhortykh, M., & Urman, A. (2022) · Journal of Information Science · doi:10.1177/01655515221093029 · Cited by 51
Li, R., Kingsley, S., Fan, C., Sinha, P., Wai, N., Lee, J., Shen, H., Eslami, M., & Hong, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3582074 · Cited by 49
Das, D., Guha, S., Brubaker, J. R., & Semaan, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642669 · Cited by 50
Reasoning about Political Bias in Content Moderation
Jiang, S., Robertson, R. E., & Wilson, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7117 · Cited by 49
Self-citationAuditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation
Robertson, R. E., Jiang, S., Lazer, D. M. J., & Wilson, C. (2019) · ACM Web Science Conference (WebSci) · doi:10.1145/3292522.3326047 · Cited by 46
Self-citationThe case for voter-centered audits of search engines during political elections
Mustafaraj, E., Lurie, E., & Devine, C. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372835 · Cited by 46
Juneja, P., Bhuiyan, M. M., & Mitra, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580846 · Cited by 44
We are the Change that we Seek: Information Interactions During a Change of Viewpoint
McKay, D., Makri, S., Lopez, M. G., MacFarlane, A., Missaoui, S., Porlezza, C., & Cooper, G. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377975 · Cited by 42
Revealing The Secret Power: How Algorithms Can Influence Content Visibility on Social Media
Galeazzi, A., Paudel, P., Conti, M., Cristofaro, E. D., & Stringhini, G. (2026) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2026.230718 · Cited by 40
Post-January 6th deplatforming reduced the reach of misinformation on Twitter
McCabe, S., Ferrari, D., Green, J., Lazer, D. M. J., & Esterling, K. (2024) · Nature · doi:10.1038/s41586-024-07524-8 · Cited by 38
Self-citationChallenges in Understanding Human-Algorithm Entanglement During Online Information Consumption
Lewandowsky, S., Robertson, R. E., & DiResta, R. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231180809 · Cited by 37
Self-citationAn investigation of biases in web search engine query suggestions
Bonart, M., Samokhina, A., Heisenberg, G., & Schaer, P. (2019) · Online Information Review · doi:10.1108/oir-11-2018-0341 · Cited by 37
How rediscovering nodality can improve democratic governance in a digital world
Margetts, H. Z., & John, P. C. (2023) · Public Administration · doi:10.1111/padm.12960 · Cited by 37
Albadi, N., Kurdi, M., & Mishra, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555618 · Cited by 36
Google the Gatekeeper: How Search Components Affect Clicks and Attention
Gleason, J. M., Hu, D., Robertson, R. E., & Wilson, C. (2023) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v17i1.22142 · Cited by 35
Self-citationEkström, A. G., Madison, G., Olsson, E. J., & Tsapos, M. (2023) · Information Communication & Society · doi:10.1080/1369118x.2023.2230242 · Cited by 35
Prabhudesai, S., Kasi, A. P., Mansingh, A., Antar, A. D., Shen, H., & Banović, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713714 · Cited by 35
Urman, A., & Makhortykh, M. (2022) · New Media & Society · doi:10.1177/14614448221099536 · Cited by 36
Wu, S., & Resnick, P. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18105 · Cited by 34
Exposure to Extremely Partisan News from the Other Political Side Shows Scarce Boomerang Effects
Casas, A., Menchen-Trevino, E., & Wojcieszak, M. (2023) · Political Behavior · doi:10.1007/s11109-021-09769-9 · Cited by 26
DeVerna, M., Pierri, F., Ahn, Y., Fortunato, S., Flammini, A., & Menczer, F. (2025) · npj Complexity · doi:10.1038/s44260-025-00038-y · Cited by 31
Wojcieszak, M., von Hohenberg, B. C., Casas, A., Menchen-Trevino, E., de Leeuw, S., Gonçalves, A., & Boon, M. (2022) · Humanities and Social Sciences Communications · doi:10.1057/s41599-022-01423-x · Cited by 32
Makhortykh, M., Urman, A., & Ulloa, R. (2021) · First Monday · doi:10.5210/fm.v26i10.11562 · Cited by 30
Auditing the representation of migrants in image web search results
Urman, A., Makhortykh, M., & Ulloa, R. (2022) · Humanities and Social Sciences Communications · doi:10.1057/s41599-022-01144-1 · Cited by 30
Ideological self-selection in online news exposure: Evidence from Europe and the US
Mangold, F., Schoch, D., & Stier, S. (2024) · Science Advances · doi:10.1126/sciadv.adg9287 · Cited by 29
Modeling and Measuring Expressed (Dis)belief in (Mis)information
Jiang, S., Metzger, M. J., Flanagin, A. J., & Wilson, C. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7302 · Cited by 29
Self-citationBuchanan, G. R., Kelly, R., Makri, S., & McKay, D. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505823 · Cited by 28
Who's the Guinea Pig? Investigating Online A/B/n Tests in-the-Wild
Jiang, S., Martin, J., & Wilson, C. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287565 · Cited by 28
Self-citationAre Search Engines Biased? Detecting and Reducing Bias using Meta Search Engines
Maillé, P., Maudet, G., Simon, M., & Tuffin, B. (2022) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2022.101132 · Cited by 27
Auditing the Biases Enacted by YouTube for Political Topics in Germany
Heuer, H., Hoch, H., Breiter, A., & Theocharis, Y. (2021) · Mensch und Computer (MuC) · doi:10.1145/3473856.3473864 · Cited by 26
Metadata
- Authors
- Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C.
- Affiliations
- Northeastern University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2018-11
- Volume
- 2
- Issue
- CSCW
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2898883664
- Cited by
- 209
- References
- 76
- Citations by year
- 2026: 18 · 2025: 27 · 2024: 33 · 2023: 31 · 2022: 30 · 2021: 41 · 2020: 13 · 2019: 15 · 2018: 1
- FWCI
- 28.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- National Science Foundation (IIS-1408345,IIS-1553088)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Communication › Social Media and Politics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- Google search, personalized search, filter bubbles, search result ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.