Auditing Partisan Audience Bias within Google Search
The DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias and personalization within Google Search. Following Donald Trump's inauguration, we recruited 187 participants to complete a survey and install a browser extension that enabled us to collect Search Engine Results Pages (SERPs) from their computers. To quantify partisan audience bias, we developed a domain-level score by leveraging the sharing propensities of registered voters on a large Twitter panel. We found little evidence for the "filter bubble'' hypothesis. Instead, we found that results positioned toward the bottom of Google SERPs were more left-leaning than results positioned toward the top, and that the direction and magnitude of overall lean varied by search query, component type (e.g. "answer boxes"), and other factors. Utilizing rank-weighted metrics that we adapted from prior work, we also found that Google's rankings shifted the average lean of SERPs to the right of their unweighted average.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research on AI auditing has focused on evaluating whether specific applications meet predefined industry requirements.”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Robertson et al. 2018
In the citing paper
“For instance, researchers have created procedures for auditing AI systems used in recruitment (Kazim et al. 2021), online search (Robertson et al. 2018), and medical diagnostics (Liu et al. 2022).”
From Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries (2024) · cites this paper as Robertson et al. 2018
In the citing paper
“For example, it is well-known that certain users on advertising platforms are more valuable to advertisers than others [48,55,65].”
From Discrimination through Optimization (2019) · cites this paper as [65]
In the citing paper
“Some researchers have leveraged narrow conceptions of auditing to test for bias and discrimination in online ad delivery (Ali et al., 2019;Sweeney, 2013) and autocomplete algorithms (Robertson et al., 2018), for fairness in image classification systems (Morina et al., 2019), for accuracy in news curation systems (Bandy & Diakopoulos, 2019), for completeness in datasets (Coston et al., 2021;Sookhak”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Robertson et al., 2018
Citations
Cites
Birds of a Feather: Homophily in Social Networks
McPherson, M., Smith-Lovin, L., & Cook, J. M. (2001) · Annual Review of Sociology · doi:10.1146/annurev.soc.27.1.415 · Cited by 18,567
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“Spurred by these concerns, research on the spread of fake news on social media has flourished [1, 51, 60, 96].”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 15×
“The resulting bias score is continuous and scales from -1 to 1, where a score of -1 means that a domain i was shared exclusively by Democrats (ri = 0, di > 0), and a score of 1 means that a domain i was shared exclusively by Republicans (ri > 0, di = 0).”
BackgroundMethodResultInfluential(2011) · Cited by 4,430
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
Self-citationProlific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Self-citationExposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 1×
“We utilized two established crowdsourcing services [72, 75] to recruit a demographically diverse sample of 187 participants to take our survey.”
Method
Research works printed but not linked: 17
- Nicholas J. Belkin. 1980. Anamolous States of Knowledge as a Basis for Information Retrieval. Canadian Journal of Information and Library Sciences 5 (1980), 133--143.
- Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
- William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: A Cross-National Survey. (2017).
- Google. 2016. Google Trends: Year in Search. Google Trends. (2016). https://trends.google.com/trends/yis/2016/ GLOBAL
- Google. 2017. Google Chrome Privacy Notice. (2017). https://www.google.com/chrome/browser/privacy/ #browser-modes Accessed: 2017-04-01. {42} Jeffrey Gottfried and Elisa Shearer. 2016. News use across social media platforms. Pew Research Center. (2016). http://www.journalism.org/2016/05/26/news-use-across-social-media-platforms-2016/
- Cecilia Kang, Tiffany Hsu, Kevin Roose, Natasha Singer, and Matthew Rosenberg. 2018. Mark Zuckerberg Testimony: Day 2 Brings Tougher Questioning. The New York Times (April 2018).
- Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.
- Amy Mitchell, Jeffrey Gottfried, Jocelyn Kiley, and Katerina Eva Matsa. 2014. Political Polarization & Media Habits. Pew Research Center's Journalism Project. (Oct. 2014).
- Nic Newman, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. SSRN Electronic Journal (2017).
- 10.5555/2029079 · doi:10.5555/2029079
- Media Insight Project. 2014. The Personal News Cycle: How Americans Choose to Get Their News. Media Insight Project. (March 2014). https://www.americanpressinstitute.org/publications/reports/survey-research/ how-americans-get-news/
- Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of "Data and Discrimination: Converting Critical Concerns into Productive Inquiry", a Productivereconference at the 64th Annual Meeting of the International Communication Association.
- Jennifer A Stark and Nicholas Diakopoulos. 2017. Using Baselines for Algorithm Audits. In European Conference on Data and Computational Journalism. 3.
- Danny Sullivan. 2016. Google Now Handles at Least 2 Trillion Searches per Year. Search Engine Land. (May 2016). https://searchengineland.com/google-now-handles-2--999-trillion-searches-per-year-250247
- New York Times. 1995. Edward Bernays, 'Father of Public Relations' And Leader in Opinion Making, Dies at 103. (1995).
- Francesca Tripodi. 2018. Searching for Alternative Facts: Analyzing Scriptural Inference in Conservative News Practices. Data & Society. (May 2018).
Cited by
Items 301–302 of 302
Jo, C. W., & Wojcieszak, M. (2026) · Journal of Communication · doi:10.1093/joc/jqag014
Wong, M. A. (2024) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Metadata
- Authors
- Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C.
- Affiliations
- Northeastern University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2018-11
- Volume
- 2
- Issue
- CSCW
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2898883664
- Cited by
- 209
- References
- 76
- Citations by year
- 2026: 18 · 2025: 27 · 2024: 33 · 2023: 31 · 2022: 30 · 2021: 41 · 2020: 13 · 2019: 15 · 2018: 1
- FWCI
- 28.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- National Science Foundation (IIS-1408345,IIS-1553088)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Communication › Social Media and Politics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- Google search, personalized search, filter bubbles, search result ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.