Measuring Price Discrimination and Steering on E-commerce Web Sites
The DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 5 of the 26 sources its claims cite were checked for support.
Abstract
Today, many e-commerce websites personalize their content, including Netflix (movie recommendations), Amazon (product suggestions), and Yelp (business reviews). In many cases, personalization provides advantages for users: for example, when a user searches for an ambiguous query such as ``router,'' Amazon may be able to suggest the woodworking tool instead of the networking device. However, personalization on e-commerce sites may also be used to the user's disadvantage by manipulating the products shown (price steering) or by customizing the prices of products (price discrimination). Unfortunately, today, we lack the tools and techniques necessary to be able to detect such behavior.
Overview
CoChat Summary
TL;DR
Hannak et al. develop a control-based measurement methodology and use 300 real-world users plus synthetic accounts to detect price steering and discrimination on 16 e-commerce sites. They find personalization on nine sites, including members-only hotel discounts, A/B-test steering, mobile-browser steering/discrimination, and history-based reordering.
Key findings
- Across 16 sites, the authors find evidence of personalization on nine sites (four general retailers and five travel sites); real-world AMT data showed price inconsistencies, e.g., a hotel priced at $565 for the control/comparison but $633 for an AMT user.
- Cheaptickets and Orbitz showed price discrimination: logged-in users received different prices on ≈5% of hotels, $12 cheaper on average, via “Members Only” reductions. Expedia and Hotels.com used A/B buckets: the clear-cookie treatment was assigned 62% to bucket 1, 21% to bucket 2, 9% to bucket 3, and 7% unknown; bucket 3 users saw >60% different results, while buckets 1 and 2 had higher nDCG values, indicating more expensive hotels near the top.
- Home Depot and Travelocity personalized for mobile devices: Home Depot served 24 results to desktop/Android but 48 to iOS, steered mobile users toward more expensive products, and discriminated against Android on ≈6% of prices with an average differential of ≈$0.41. Travelocity lowered prices by ≈$15 on ≈5% of hotels for iOS users.
- Priceline altered hotel search results based on click and purchase history, but the reordering was not correlated with prices and no price discrimination was observed.
Methods
- Real-world data: 300 Amazon Mechanical Turk users (100 each for retail, hotel, and car experiments) ran searches through a PAC proxy and iframe, forwarding their cookies; the study covered 16 sites (10 general retailers, 6 travel retailers) with 20 searches per site, plus PhantomJS comparison/control searches from the same IP and static DNS.
- Synthetic experiments: PhantomJS instances with an n+1 control design tested account/cookies, OS/browser User-Agent, and click/purchase history; each ran 20 queries per site per day, spaced one minute apart, with travel purchase history built via no-payment reservations that were later canceled.
- Analysis used Jaccard index, Kendall’s τ, nDCG, percent of products with inconsistent prices, and average price difference, comparing treatments against controls to separate noise from personalization.
Limitations
- The paper states the study is limited to US e-commerce sites, US IP addresses, English searches, and US users; it can identify only positive instances of personalization, not absence, and leaves other features such as geolocation, HTTP Referer, browsing history, and purchase history to future work. It also notes unexplained personalization on Newegg and Sears and that experiments were conducted in spring 2014, so results may not hold in the future.
- The paper states that Expedia and Hotels.com could not be analyzed from the AMT experiment because A/B testing made comparison and control results rarely match; rental car steering was not examined because travel sites present cars in a deterministic grid; and the authors could not infer which specific AMT user characteristics triggered personalization.
- The text makes evident that Home Depot’s differences may be artifacts of different server-side implementations for desktop and mobile rather than an explicit personalization algorithm, though they still qualify under the paper’s definition; synthetic purchase-history treatments may not embody realistic user behavior but were designed as unambiguous signals.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 12 Not checked · 107 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We develop a measurement infrastructure that is able to distinguish genuine personalization of e-commerce sites from other sources of noise; this methodology is based on previous work on measuring personalization of web search services [17].”
What the sources say
- Supportedcites [17] [17] A. Hannak, P. Sapiezy´ nski, A. M. Kakhki, B. Krishnamurthy, D. Lazer, A. Mislove, and C. Wilson. Measuring Personalization of Web Search. WWW , 2013. - source
The citation supports the methodological precedent, not the citing authors’ claim that they developed an e-commerce measurement infrastructure. The source’s Concluding Discussion (§7) describes a methodology for measuring web-search personalization that controls for sources of noise; its Introduction (§1) says the methodology can generalize to product search. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find evidence for some form of personalization on nine of these e-commerce sites.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To summarize our findings in this section: we find evidence for price steering and price discrimination on four general retailers and five travel sites.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, these four sites do exhibit significant noise: Kendall's τ is ≤ 0.83 in all four cases.”
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, we find no correlation between browser/OS choice and high personalization.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
66 more claims
Create a free account to see all 126 claims and to verify the 5 references still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 35
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, it is difficult to discover what the actual reason for a price fluctuation is and enforce accordingly -if a price fluctuation is even observed at all (Mikians et al., 2012;Hannak et. al., 2014).”
From Discrimination grounds and personalised pricing: Consumer perceptions of fairness, norm alignment, legality, and trust in markets (2024) · cites this paper as Hannak et. al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data are being leveraged to make inferences about how much to pay for a product (Hannak et al., 2014), about the likelihood of being a terrorist or about users health (Yom-Tov, 2016) and employability (Rosenblat et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Hannak et al., 2014
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Whenever the confidence intervals around these fractions for two audiences are non-overlapping, we can make a claim that the gender or racial makeups of two audiences are significantly different [17].”
From Discrimination through Optimization (2019) · cites this paper as [17]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the converse is not true: overlapping confidence intervals do not necessarily mean that the means are not different (see Figure 4 in [17] for explanation).”
From Discrimination through Optimization (2019) · cites this paper as [17]
Experimental — accuracy not yet measuredIn the citing paper
“Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol [36], and in e-commerce platforms [35].”
From The fallacy of inscrutability (2018) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1. INTRODUCTION · p. 2
“Our real-world data indicates that eight of these sites implement personalization”
§ 4.5 Summary · p. 7
“we find evidence for price steering and price discrimination on four general retailers and five travel sites.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“prices fall by ≈ $15 on ≈ 5% of hotels (or 5 out 50 per page) for iOS users.”
Reported 5 · recomputed 10
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2 Implementation · p. 4
“| Staples | http://sears.com | Office supplies |”
§ 3.2 Implementation · p. 4
“| Sears | http://sears.com | Clothes, housewares |”
Citations
Cites
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
A NEW MEASURE OF RANK CORRELATION
KENDALL, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,269
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
(2011) · Cited by 4,430
Information Rules: A Strategic Guide to the Network Economy
Shaw, M. J., Shapiro, C., & Varian, H. R. (2000) · The Academy of Management Review · doi:10.2307/259025 · Cited by 4,124
Becker, W. E., Shapiro, C., & Varian, H. R. (1999) · The Journal of Economic Education · doi:10.1080/00220489909595956 · Cited by 2,685
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Information Rules: A Strategic Guide to the Network Economy
Becker, W. E., Shapiro, C., & Varian, H. R. (1999) · The Journal of Economic Education · doi:10.2307/1183273 · Cited by 2,557
The proof and measurement of association between two things
Spearman, C. (2010) · International Journal of Epidemiology · doi:10.1093/ije/dyq191 · Cited by 1,681
Google news personalization: scalable online collaborative filtering
Das, A. S., Datar, M., Garg, A., & Rajaram, S. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242610 · Cited by 1,511
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,444
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2000) · Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/345508.345545 · Cited by 967
(2003) · doi:10.5555/644108.644113 · Cited by 762
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
Ordinal Measures of Association
Kruskal, W. H. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501481 · Cited by 740
Ghose, A., Ipeirotis, P. G., & Li, B. (2012) · Marketing Science · doi:10.1287/mksc.1110.0700 · Cited by 649
Spearman's Footrule as a Measure of Disarray
Diaconis, P., & Graham, R. L. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01624.x · Cited by 640
Fagin, R., Kumar, R., & Sivakumar, D. (2003) · SIAM Journal on Discrete Mathematics · doi:10.1137/s0895480102412856 · Cited by 552
A large-scale evaluation and analysis of personalized search strategies
Dou, Z., Song, R., & Wen, J.-R. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242651 · Cited by 530
(2009) · Wiley eBooks · doi:10.1002/9780470744734 · Cited by 413
Measuring personalization of web search
Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488435 · Cited by 396
Self-citation24. How Companies Learn Your Secrets
Duhigg, C. (2013) · The Best Business Writing 2013 · doi:10.7312/star16075-025 · Cited by 362
Pitkow, J., Schütze, H., Cass, T., Cooley, R., Turnbull, D., Edmonds, A., Adar, E., & Breuel, T. (2002) · Communications of the ACM · doi:10.1145/567498.567526 · Cited by 327
Generalized distances between rankings
Kumar, R., & Vassilvitskii, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772749 · Cited by 273
Calo, M. R. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2309703 · Cited by 265
Demographic prediction based on user's browsing behavior
Hu, J., Zeng, H.-J., Li, H., Niu, C., & Chen, Z. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242594 · Cited by 259
Detecting price and search discrimination on the internet
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2012) · Proceedings of the 11th ACM Workshop on Hot Topics in Networks · doi:10.1145/2390231.2390245 · Cited by 244
Mining long-term search history to improve search accuracy
Tan, B., Shen, X., & Zhai, C. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150493 · Cited by 234
Challenges in measuring online advertising systems
Guha, S., Cheng, B., & Francis, P. (2010) · Proceedings of the 10th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1879141.1879152 · Cited by 169
Ghose, A., Ipeirotis, P. G., & Li, B. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1856558 · Cited by 162
Privacy-enhancing personalized web search
Xu, Y., Wang, K., Zhang, B., & Chen, Z. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242652 · Cited by 149
Rank Aggregation for Similar Items
Sculley, D. (2007) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972771.66 · Cited by 107
Who uses web search for what: and how
Weber, I., & Jaimes, A. (2011) · Proceedings of the fourth ACM international conference on Web search and data mining · doi:10.1145/1935826.1935839 · Cited by 91
Crowd-assisted search for price discrimination in e-commerce: first results
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535415 · Cited by 82
Understanding what they do with what they know
Wills, C. E., & Tatar, C. (2012) · Proceedings of the 2012 ACM workshop on Privacy in the electronic society · doi:10.1145/2381966.2381969 · Cited by 79
(2014) · Lirias · Cited by 61
Relevance Ranking for Geographic IR
Andrade, L., & Silva, M. J. (2006) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 59
Exposing Inconsistent Web Search Results with Bobble
Xing, X., Meng, W., Doozan, D., Feamster, N., Lee, W., & Snoeren, A. C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-04918-2_13 · Cited by 56
An approach to controlling user models and personalization effects in recommender systems
Bakalov, F., Meurs, M.-J., König-Ries, B., Sateli, B., Witte, R., Butler, G., & Tsang, A. (2013) · Proceedings of the 2013 international conference on Intelligent user interfaces · doi:10.1145/2449396.2449405 · Cited by 55
Discovering users' specific geo intention in web search
Yi, X., Raghavan, H., & Leggetter, C. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526774 · Cited by 54
A query-aware document ranking method for geographic information retrieval
Yu, B., & Cai, G. (2007) · Workshop on Geographic Information Retrieval (GIR) · doi:10.1145/1316948.1316962 · Cited by 51
Take this personally: pollution attacks on personalized services
Xing, X., Meng, W., Doozan, D., Snoeren, A. C., Feamster, N., & Lee, W. (2013) · Cited by 41
Know your personalization: learning topic level personalization in online services
Majumder, A., & Shrivastava, N. (2013) · International World Wide Web Conference (WWW) · doi:10.1145/2488388.2488464 · Cited by 39
Crowd-assisted Search for Price Discrimination in E-Commerce: First resultsPreprint
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.4531 · Cited by 28
RANK TESTS FOR INDEPENDENCE — WITH A WEIGHTED CONTAMINATION ALTERNATIVE
Shieh, G. S., & Bai, Z. (2000) · National University of Singapore · Cited by 27
Economics and Management of the Food Industry
Dorfman, J. H. (2014) · Routledge eBooks · doi:10.4324/9780203795736 · Cited by 8
Know Your Personalization: Learning Topic level Personalization in Online ServicesPreprint
Majumder, A., & Shrivastava, N. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1212.3390 · Cited by 7
Png, I. (2013) · Routledge eBooks · doi:10.4324/9780203116098 · Cited by 5
Research works printed but not linked: 1
- T. Wadhwa . How Advertisers Can Use Your Personal Information to Make You Pay Higher Prices. Huffington Post , 2014 . http://www.huffingtonpost.com/tarun-wadhwa/how-advertisers-can-use-y_b_4703013.html. T. Wadhwa. How Advertisers Can Use Your Personal Information to Make You Pay Higher Prices. Huffington Post, 2014. http://www.huffingtonpost.com/tarun-wadhwa/how-advertisers-can-use-y_b_4703013.html.
Cited by
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“Social data are being leveraged to make inferences about how much to pay for a product (Hannak et al., 2014), about the likelihood of being a terrorist or about users health (Yom-Tov, 2016) and employability (Rosenblat et al., 2014).”
BackgroundOnline Tracking: A 1-million-site Measurement and Analysis
Englehardt, S., & Narayanan, A. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978313 · Cited by 647
“Many other practices that raise privacy or ethical concerns have been studied: price discrimination, where a site shows different prices to different consumers for the same product [19, 63]; steering, a gentler form of price discrimination where a product search shows differently-priced results for different users [32…”
BackgroundAuditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580
“Previous large scale algorithmic auditing include measuring discrimination on AirBnB [14], personalization onweb search [19] and price discrimination on e-commerce web sites [20].”
BackgroundMathur, A., Kshirsagar, M., & Mayer, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445610 · Cited by 331
“know why they are shown a limited set of choices [23].”
Background
Metadata
- Authors
- Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C.
- Affiliations
- Northeastern University, Boston, MA, USA
- Northeastern University (US)
- Venue
- Proceedings of the 2014 Conference on Internet Measurement Conference
- Publisher
- ACM
- Published
- 2014-11-05
- Pages
- 305-318
- OpenAlex
- W2161392727
- Cited by
- 405
- References
- 51
- Citations by year
- 2026: 30 · 2025: 57 · 2024: 25 · 2023: 46 · 2022: 36 · 2021: 41 · 2020: 28 · 2019: 34 · 2018: 40 · 2017: 28 · 2016: 25 · 2015: 14 · 2014: 1
- FWCI
- 63.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Internet Measurement Conference (IMC) (publishedVersion)
- Funders
- National Science Foundation (CHS- 1408345)
- Division of Computer and Network Systems (CNS-1054233)
- Amazon Web Services in Education
- Army Research Office (W911NF-12-1-0556)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- price discrimination, e-commerce personalization, recommender systems, personalized pricing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.