Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Search engines are a primary means through which people obtain information in today»s connected world. Yet, apart from the search engine companies themselves, little is known about how their algorithms filter, rank, and present the web to users. This question is especially pertinent with respect to political queries, given growing concerns about filter bubbles, and the recent finding that bias or favoritism in search rankings can influence voting behavior. In this study, we conduct a targeted algorithm audit of Google Search using a dynamic set of political queries. We designed a Chrome extension to survey participants and collect the Search Engine Results Pages (SERPs) and autocomplete suggestions that they would have been exposed to while searching our set of political queries during the month after Donald Trump»s Presidential inauguration. Using this data, we found significant differences in the composition and personalization of politically-related SERPs by query type, subjects» characteristics, and date.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“Future work on misinformation may want to consider how search engines and highly ranked components featuring direct answers ( knowledge and people-ask ) and news or social media ( twitter , news-card , and news-triplets ) might contribute to the spread of untrue information [2, 30, 39, 44, 73].”
BackgroundSocial Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 2×
“Future work on misinformation may want to consider how search engines and highly ranked components featuring direct answers ( knowledge and people-ask ) and news or social media ( twitter , news-card , and news-triplets ) might contribute to the spread of untrue information [2, 30, 39, 44, 73].”
BackgroundLazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160 · cited 1×
“Future work on misinformation may want to consider how search engines and highly ranked components featuring direct answers ( knowledge and people-ask ) and news or social media ( twitter , news-card , and news-triplets ) might contribute to the spread of untrue information [2, 30, 39, 44, 73].”
BackgroundSelf-citationEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 1×
“Recent research suggests that participants recruited from these platforms are more naïve and less dishonest than participants recruited on AMT [64].”
BackgroundMemory: A Contribution to Experimental Psychology
Ebbinghaus, H. (2013) · Annals of Neurosciences · doi:10.5214/ans.0972.7531.200408 · Cited by 2,827
Reuters Institute Digital News Report 2015
Newman, N., Levy, D. A. L., & Nielsen, R. K. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2619576 · Cited by 1,890
FILTER BUBBLE: Wie wir im Internet entmündigt werden
Pariser, E. (2012) · doi:10.3139/9783446431164 · Cited by 1,849
The serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“These heuristic-driven behavioral patterns, known as order effects , are among the most robust effects ever discovered in the psychological and behavioral sciences, and imbue the entity ranking the content with the power to change attitudes, beliefs, and behavior [3, 20, 21, 34, 55].”
BackgroundMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 1×
“On the inauguration, the day after, and once a week following, we posted recruitment ads on CrowdFlower (http://crowdflower. com) and Prolific Academic (http://prolific.ac), online subject pools that are comparable to the widely used Amazon Mechanical Turk (AMT; http://mturk.com) [6, 9, 65].”
MethodThe Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Inoculating the Public against Misinformation about Climate Change
van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017) · Global Challenges · doi:10.1002/gch2.201600008 · Cited by 1,058 · cited 1×
“Future research could incorporate various types of user studies with our data collection method to potentially answer these questions [1, 21, 59, 60, 72], and should investigate possibilities for suppressing unwanted influences with design interventions [22, 42, 54].”
BackgroundIn Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · cited 6×
“…in the study of misinformation or fake news have thus far focused primarily on social media, our results point out a diverse ecosystem of information presentation that, in combination with the trust placed in search engines [7, 60], could increase exposure to and consumption of mis-information.”
BackgroundResultInfluentialMeasuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405 · cited 1×
“Utilizing this paradigm, researchers have conducted audits on algorithms in online marketplaces [12, 13, 33, 47, 48], search ranking bias on Twitter [38], user awareness of Facebook’s NewsFeed al-gorithm [24], localization of online maps [69], and, most relevant here, search engine rankings [32,…”
MethodSelf-citationMeasuring personalization of web search
Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488435 · Cited by 396
Self-citationAccountability in a computerized society
Nissenbaum, H. (1996) · Science and Engineering Ethics · doi:10.1007/bf02639315 · Cited by 346
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · cited 1×
“Utilizing this paradigm, researchers have conducted audits on algorithms in online marketplaces [12, 13, 33, 47, 48], search ranking bias on Twitter [38], user awareness of Facebook’s NewsFeed al-gorithm [24], localization of online maps [69], and, most relevant here, search engine rankings [32,…”
MethodLeveraging position bias to improve peer recommendation
Lerman, K., & Hogg, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098914 · Cited by 96 · cited 1×
“Similar browsing patterns appear to occur on virtually any online platform where content is ranked [4, 19, 23, 31, 40].”
BackgroundCrowd-assisted search for price discrimination in e-commerce: first results
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535415 · Cited by 82
Cited by
Auditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209
Self-citationSun, L., Wei, M., Sun, Y., Suh, Y. J., Shen, L., & Yang, S. (2023) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmad045 · Cited by 146
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133
Self-citationEvaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
Algorithmic Political Bias in Artificial Intelligence Systems
Peters, U. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00512-8 · Cited by 111
“Markers of political orientation might become predictors for relevant information for a given individual (Robertson et al., 2018) and be used by personalization algorithms to selectively present some contents (or people and groups) to that individual and ignore others because of their political…”
Background
Metadata
- Authors
- Robertson, R. E., Lazer, D., & Wilson, C.
- Venue
- Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18
- Publisher
- ACM Press
- Published
- 2018
- Pages
- 955-965
- OpenAlex
- W2788848737
- Cited by
- 139
- References
- 23
- FWCI
- 13.35
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.