Measuring personalization of web search
Open accessAbstract
Web search is an integral part of our daily lives. Recently, there has been a trend of personalization in Web search, where different users receive different results for the same search query. The increasing personalization is leading to concerns about Filter Bubble effects, where certain users are simply unable to access information that the search engines' algorithm decides is irrelevant. Despite these concerns, there has been little quantification of the extent of personalization in Web search today, or the user attributes that cause it.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Static DNS entries direct all query traffic to specific IP addresses of the retailers to eliminate errors arising from differences between datacenters.
“Second , we use static DNS entries to direct all of our query traffic to specific IP addresses of the retailers. This eliminates errors arising from differences between datacenters.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [17]
In “Controlling Against Noise,” the paper says it used static DNS entries to send query traffic to a specific Web search provider IP address to eliminate differences between datacenters. It does not describe directing traffic to retailers’ IP addresses; that changes the target of the method. (LLM, conf=0.90)
Experimental — accuracy not yet measuredRecommender algorithms installed on online platforms help users efficiently access the information they are looking for while also reducing information overload.
“Recommender algorithms installed on online platforms help users efficiently access the information they are looking for while also reducing information overload (Hannak et al., 2013;Lazer, 2015).”
From Do Search Algorithms Endanger Democracy? An Experimental Investigation of Algorithm Effects on Political Polarization (2020) · cites this paper as Hannak et al., 2013
The Introduction describes personalized search as ranking webpages a user is likely to be interested in more highly, which supports the information-finding part of the claim in principle. But the Recommendations section says the study did not determine whether personalization improved users’ search experiences, and the passages do not establish that it reduces information overload. (LLM, conf=0.83)
Experimental — accuracy not yet measuredAll searches for a given query are executed at the same time to eliminate differences in results due to temporal effects.
“first , all searches for a given query are executed at the same time. This eliminates differences in results due to temporal effects.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [17]
The paper develops a measurement infrastructure able to distinguish genuine personalization of e-commerce sites from other sources of noise, based on previous work on measuring personalization of web search services.
“We develop a measurement infrastructure that is able to distinguish genuine personalization of e-commerce sites from other sources of noise; this methodology is based on previous work on measuring personalization of web search services [17].”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [17]
SSH tunnels forward the traffic of each treatment to a unique IP address in a /24 subnet to ensure that any effects due to IP geolocation affect all results equally.
“Third , although all PhantomJS instances execute on one machine, we use SSH tunnels to forward the traffic of each treatment to a unique IP address in a /24 subnet. This process ensures that any effects due to IP geolocation will affect all results equally.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [17]
On average, about 12% of Google search results exhibit differences due to personalization.
“Hannak et al. (2013) observe that, on average, about 12% of Google search results exhibit differences due to personalization.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Hannak et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
FILTER BUBBLE: Wie wir im Internet entmündigt werden
Pariser, E. (2012) · doi:10.3139/9783446431164 · Cited by 1,849 · see citing works
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · see citing works
What are you looking for?: an eye-tracking study of information usage in web search
Cutrell, E., & Guan, Z. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240690 · Cited by 460 · see citing works
Demographic prediction based on user's browsing behavior
Hu, J., Zeng, H.-J., Li, H., Niu, C., & Chen, Z. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242594 · Cited by 259 · see citing works
Flagged first, then by citations
Research works printed but not linked: 38
- Jesper Andersen, Anders Giversen, Allan H. Jensen, Rune S. Larsen, Torben Bach Pedersen, and Janne Skyt. 2000. Analyzing Clickstreams Using Subsessions, In ACM International Workshop On Data Warehousing and OLAP. ACM International Workshop On Data Warehousing and OLAP (November 2000), 25-32.
- Leonardo Andrade and Mário J. Silva. 2006. Relevance Ranking for Geographic IR. In ACM Workshop On Geographic Information Retrieval. Seattle, Washington.
- Judit Bar-Ilan, Kevin Keenoy, Eti Yaari, and Mark Levene. 2007. User rankings of search engine results. Journal of the American Society for Information Science and Technology 58, 9 (July 2007), 1254-1266.
- Andrei Broder. 2002. A Taxonomy of Web Search. ACM SIGIR Forum 36, 2 (2002), 3-10.
- Zhicong Cheng, Bin Gao, and Tie-Yan Liu. 2010. Actively Predicting Diverse Search Intent from User Browsing Behaviors. In International World Wide Web Conference. Raleigh, North Carolina.
- Frank Crown and Michael L. Nelson. 2007. Agreeing to Disagree: Search Engines and Their Public Interfaces. In ACM/IEEE-CS Joint Conference on Digital Libraries. Vancouver, Canada.
- Abhinandan Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google News Personalization: Scalable Online Collaborative Filtering. In International World Wide Web Conference. Banff, Canada.
- Zhicheng Dou, Ruihua Song, and Ji-Rong Wen. 2007. A Large-scale Evaluation and Analysis of Personalized Search Strategies. In International World Wide Web Conference. Banff, Canada.
- Weiguo Fan, Michael D. Gordon, and Praveen Pathak. 2000. Personalization of Search Engine Services for Effective Retrieval and Knowledge Management. In Annual Conference on Information Systems. Atlanta, Georgia.
- Susan Gauch, Jason Chaffee, and Alexander Pretschner. 2003. Ontology-based personalized search and browsing. Web Intelligence and Agent Systems 1 (2003), 1-3.
- Sharad Goel, Jake M. Hofman, and M Irmak Sirer. 2012. Who Does What on the Web: A Large-scale Study of Browsing Behavior. In International AAAI Conference on Weblogs and Social Media. Dublin, Ireland.
- Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-tracking analysis of user behavior in WWW search. In Conference of the ACM Special Interest Group on Information Retrieval. Sheffield, United Kingdom.
- Zhiwei Guan and Edward Cutrell. 2007. An eye tracking study of the effect of target rank on web search. In Annual Conference of the ACM Special Interest Group on Computer Human Interaction. San Jose, California.
- Saikat Guha, Bin Cheng, and Paul Francis. 2010. Challenges in Measuring Online Advertising Systems. In ACM/USENIX Internet Measurement Conference. Melbourne, Victoria, Australia.
- Aniko Hannak, Piotr Sapieżyński, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In International World Wide Web Conference. Rio de Janeiro, Brazil.
- Fang Liu, Clement Yu, and Weiyi Meng. 2002. Personalized web search by mapping user queries to categories. In ACM International Conference on Information and Knowledge Management. McLean, Virginia.
- Fang Liu, Clement Yu, and Weiyi Meng. 2004. Personalized Web Search for Improving Retrieval Effectiveness. IEEE Transactions on Knowledge and Data Engineering 16, 1 (January 2004), 28-40.
- Lori Lorigo, Bing Pan, Helene Hembrooke, Thorsten Joachims, Laura Granka, and Geri Gay. 2006. The influence of task and gender on search and evaluation behavior using Google. Information Processing and Management 42, 4 (2006), 1123-1131.
- Anirban Majumder and Nisheeth Shrivastava. 2013. Know your personalization: learning topic level personalization in online services. In International World Wide Web Conference. Rio de Janeiro, Brazil.
- Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2012. Detecting Price and Search Discrimination on the Internet. In Workshop on Hot Topics in Networks. Seattle, Washington.
- Bamshad Mobasher, Robert Cooley, and Jaideep Srivastava. 2000. Automatic Personalization Based on Web Usage Mining. Commun. ACM 43, 8 (2000), 142-151.
- Eli Pariser. 2011. The Filter Bubble: What the Internet is Hiding from You. Penguin Press.
- James Pitkow, Hinrich Schütze, Todd Cass, Rob Cooley, Don Turnbull, Andy Edmonds, Eytan Adar, and Thomas Breuel. 2002. Personalized search. Commun. ACM 45, 9 (September 2002), 50-55.
- Alexander Pretschner and Susan Gauch. 1999. Ontology based personalized search. In IEEE International Conference on Tools with Artificial Intelligence. Chicago, Illinois.
- Feng Qiu and Junghoo Cho. 2006. Automatic Identification of User Interest for Personalized Search. In International World Wide Web Conference. Edinburgh, Scotland.
- Franziska Roesner, Tadayoshi Kohno, and David Wetherall. 2012. Detecting and Defending Against Third-Party Tracking on the Web. In Symposium on Networked System Design and Implementation. San Jose, California.
- Xuehua Shen, Bin Tan, and ChengXiang Zhai. 2005. Implicit user modeling for personalized search. In ACM International Conference on Information and Knowledge Management. Bremen, Germany.
- Yelong Shen, Jun Yan, Shuicheng Yan, Lei Ji, Ning Liu, and Zheng Chen. 2011. Sparse Hidden-Dynamics Conditional Random Fields for User Intent Understanding. In International World Wide Web Conference. Hyderabad, India.
- Ahu Sieg, Bamshad Mobasher, and Robin Burke. 2007. Web Search Personalization with Ontological User Profiles. In ACM International Conference on Information and Knowledge Management.
- Jian-Tao Sun, Hua-Jun Zeng, Huan Liu, Yuchang Lu, and Zheng Chen. 2005. CubeSVD: A Novel Approach to Personalized Web Search. In International World Wide Web Conference. Chiba, Japan.
- Mingxuan Sun, Guy Lebanon, and Kevyn Collins-Thompson. 2010. Visualizing Differences in Web Search Algorithms using the Expected Weighted Hoeffding Distance. In International World Wide Web Conference. Raleigh, North Carolina.
- Bin Tan, Xuehua Shen, and ChengXiang Zhai. 2006. Mining long-term search history to improve search accuracy. In ACM SIGKDD International Conference of Knowledge Discovery and Data Mining. Philadelphia, Pennsylvania.
- Jamie Teevan, Susan T. Dumais, and Eric Horvitz. 2005. Personalizing search via automated analysis of interests and activities. In Conference of the ACM Special Interest Group on Information Retrieval. Sheffield, United Kingdom.
- Ryen W. White. 2013. Beliefs and Biases in Web Search. In Conference of the ACM Special Interest Group on Information Retrieval.
- Craig E. Wills and Can Tatar. 2012. Understanding What They Do with What They Know. In Workshop on Privacy in the Electronic Society.
- Yabo Xu, Benyu Zhang, Zheng Chen, and Ke Wang. 2007. Privacy-Enhancing Personalized Web Search. In International World Wide Web Conference. Banff, Canada.
- Xing Yi, Hema Raghavan, and Chris Leggetter. 2009. Discovering Users' Specific Geo Intention in Web Search. In International World Wide Web Conference. Madrid, Spain.
- Bo Yu and Guoray Cai. 2007. A query-aware document ranking method for geographic information retrieval. In ACM Workshop On Geographic Information Retrieval. Lisbon, Portugal.
Cited by
Sorted by citations
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · see citing works
Self-citationFilter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Algorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
Should we worry about filter bubbles?
Zuiderveen Borgesius, F. J., Trilling, D., Möller, J., Bodó, B., de Vreese, C. H., & Helberger, N. (2016) · Internet Policy Review · doi:10.14763/2016.1.401 · Cited by 541 · see citing works
Auditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580 · see citing works
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 368 · see citing works
Self-citationBias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · see citing works
Self-citationAuditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209 · see citing works
Self-citationProblematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · see citing works
Cho, J., Ahmed, S., Hilbert, M., Liu, B., & Luu, J. (2020) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2020.1757365 · Cited by 164 · see citing works
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
Self-citationQuantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · see citing works
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · see citing works
Self-citationThe rise of the social algorithm
Lazer, D. (2015) · Science · doi:10.1126/science.aab1422 · Cited by 133 · see citing works
Self-citationUsers choose to engage with more partisan news than they are exposed to on Google Search
Robertson, R. E., Green, J., Ruck, D. J., Ognyanova, K., Wilson, C., & Lazer, D. (2023) · Nature · doi:10.1038/s41586-023-06078-5 · Cited by 126 · see citing works
Self-citationSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
Crowd-assisted search for price discrimination in e-commerce: first results
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535415 · Cited by 82 · see citing works
Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems
Kroll, J. A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445937 · Cited by 2 · see citing works
- Authors
- Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C.
- Affiliations
- Northeastern University, Boston, MA, USA
- Technical University of Denmark, Copenhagen, Denmark
- AT&T (United States) (US)
- Northeastern University (US)
- Venue
- Proceedings of the 22nd international conference on World Wide Web
- Publisher
- ACM
- Published
- 2013-05-13
- Pages
- 527-538
- OpenAlex
- W2109882182
- Cited by
- 396
- References
- 4
- Citations by year
- 2026: 10 · 2025: 19 · 2024: 33 · 2023: 24 · 2022: 39 · 2021: 55 · 2020: 34 · 2019: 38 · 2018: 33 · 2017: 31 · 2016: 32 · 2015: 29 · 2014: 15 · 2013: 3
- FWCI
- 21.16
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International World Wide Web Conference (WWW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1054233, 1408345, 0964465)
- Amazon Web Services
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Keywords
- web search personalization, filter bubbles, personalized search, user characteristics, search engine algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
