CoChat

Auditing Partisan Audience Bias within Google Search

Robertson, Ronald E., Jiang, Shan, Joseph, Kenneth, Friedland, Lisa, Lazer, David, Wilson, Christo · 2018 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 76 of the 77 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias and personalization within Google Search. Following Donald Trump's inauguration, we recruited 187 participants to complete a survey and install a browser extension that enabled us to collect Search Engine Results Pages (SERPs) from their computers. To quantify partisan audience bias, we developed a domain-level score by leveraging the sharing propensities of registered voters on a large Twitter panel. We found little evidence for the "filter bubble'' hypothesis. Instead, we found that results positioned toward the bottom of Google SERPs were more left-leaning than results positioned toward the top, and that the direction and magnitude of overall lean varied by search query, component type (e.g. "answer boxes"), and other factors. Utilizing rank-weighted metrics that we adapted from prior work, we also found that Google's rankings shifted the average lean of SERPs to the right of their unweighted average.

Overview

Integrity signals

Registry

Crossref · 10.1145/3274417

Cites

76 of 77 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 77 references

Research works printed but not linked: 17

  1. Nicholas J. Belkin. 1980. Anamolous States of Knowledge as a Basis for Information Retrieval. Canadian Journal of Information and Library Sciences 5 (1980), 133--143.
  2. Nicholas Diakopoulos Daniel Trielli Jennifer Stark and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google Amazon Facebook and Apple M. Moore and D. Tambini (Eds.). 22.
  3. William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: A Cross-National Survey. (2017).
  4. Google. 2016. Google Trends: Year in Search. Google Trends. (2016). https://trends.google.com/trends/yis/2016/ GLOBAL
  5. Google. 2017. Google Chrome Privacy Notice. (2017). https://www.google.com/chrome/browser/privacy/ #browser-modes Accessed: 2017-04-01. {42} Jeffrey Gottfried and Elisa Shearer. 2016. News use across social media platforms. Pew Research Center. (2016). http://www.journalism.org/2016/05/26/news-use-across-social-media-platforms-2016/
  6. Cecilia Kang, Tiffany Hsu, Kevin Roose, Natasha Singer, and Matthew Rosenberg. 2018. Mark Zuckerberg Testimony: Day 2 Brings Tougher Questioning. The New York Times (April 2018).
  7. Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.
  8. Amy Mitchell, Jeffrey Gottfried, Jocelyn Kiley, and Katerina Eva Matsa. 2014. Political Polarization & Media Habits. Pew Research Center's Journalism Project. (Oct. 2014).
  9. Nic Newman, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. SSRN Electronic Journal (2017).
  10. 10.5555/2029079 · doi:10.5555/2029079
  11. Media Insight Project. 2014. The Personal News Cycle: How Americans Choose to Get Their News. Media Insight Project. (March 2014). https://www.americanpressinstitute.org/publications/reports/survey-research/ how-americans-get-news/
  12. Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.
  13. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of "Data and Discrimination: Converting Critical Concerns into Productive Inquiry", a Productivereconference at the 64th Annual Meeting of the International Communication Association.
  14. Jennifer A Stark and Nicholas Diakopoulos. 2017. Using Baselines for Algorithm Audits. In European Conference on Data and Computational Journalism. 3.
  15. Danny Sullivan. 2016. Google Now Handles at Least 2 Trillion Searches per Year. Search Engine Land. (May 2016). https://searchengineland.com/google-now-handles-2--999-trillion-searches-per-year-250247
  16. New York Times. 1995. Edward Bernays, 'Father of Public Relations' And Leader in Opinion Making, Dies at 103. (1995).
  17. Francesca Tripodi. 2018. Searching for Alternative Facts: Analyzing Scriptural Inference in Conservative News Practices. Data & Society. (May 2018).

Cited by

Metadata

Authors
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C.
Affiliations
Northeastern University (US)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2018-11
Volume
2
Issue
CSCW
Pages
1-22
ISSN
2573-0142 (ISSN-L)
OpenAlex
W2898883664
Cited by
209
References
76
Citations by year
2026: 18 · 2025: 27 · 2024: 33 · 2023: 31 · 2022: 30 · 2021: 41 · 2020: 13 · 2019: 15 · 2018: 1
FWCI
28.16
Type
journal article
Language
en
Open access
closed
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
Funders
National Science Foundation (IIS-1408345,IIS-1553088)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Communication › Social Media and Politics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
Google search, personalized search, filter bubbles, search result ranking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.