The DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 5 of the 26 sources its claims cite were checked for support.
Measuring Price Discrimination and Steering on E-commerce Web Sites
Abstract
Today, many e-commerce websites personalize their content, including Netflix (movie recommendations), Amazon (product suggestions), and Yelp (business reviews). In many cases, personalization provides advantages for users: for example, when a user searches for an ambiguous query such as ``router,'' Amazon may be able to suggest the woodworking tool instead of the networking device. However, personalization on e-commerce sites may also be used to the user's disadvantage by manipulating the products shown (price steering) or by customizing the prices of products (price discrimination). Unfortunately, today, we lack the tools and techniques necessary to be able to detect such behavior.
Overview
CoChat Summary
TL;DR
Hannak et al. develop a control-based measurement methodology and use 300 real-world users plus synthetic accounts to detect price steering and discrimination on 16 e-commerce sites. They find personalization on nine sites, including members-only hotel discounts, A/B-test steering, mobile-browser steering/discrimination, and history-based reordering.
Key findings
- Across 16 sites, the authors find evidence of personalization on nine sites (four general retailers and five travel sites); real-world AMT data showed price inconsistencies, e.g., a hotel priced at $565 for the control/comparison but $633 for an AMT user.
- Cheaptickets and Orbitz showed price discrimination: logged-in users received different prices on ≈5% of hotels, $12 cheaper on average, via “Members Only” reductions. Expedia and Hotels.com used A/B buckets: the clear-cookie treatment was assigned 62% to bucket 1, 21% to bucket 2, 9% to bucket 3, and 7% unknown; bucket 3 users saw >60% different results, while buckets 1 and 2 had higher nDCG values, indicating more expensive hotels near the top.
- Home Depot and Travelocity personalized for mobile devices: Home Depot served 24 results to desktop/Android but 48 to iOS, steered mobile users toward more expensive products, and discriminated against Android on ≈6% of prices with an average differential of ≈$0.41. Travelocity lowered prices by ≈$15 on ≈5% of hotels for iOS users.
- Priceline altered hotel search results based on click and purchase history, but the reordering was not correlated with prices and no price discrimination was observed.
Methods
- Real-world data: 300 Amazon Mechanical Turk users (100 each for retail, hotel, and car experiments) ran searches through a PAC proxy and iframe, forwarding their cookies; the study covered 16 sites (10 general retailers, 6 travel retailers) with 20 searches per site, plus PhantomJS comparison/control searches from the same IP and static DNS.
- Synthetic experiments: PhantomJS instances with an n+1 control design tested account/cookies, OS/browser User-Agent, and click/purchase history; each ran 20 queries per site per day, spaced one minute apart, with travel purchase history built via no-payment reservations that were later canceled.
- Analysis used Jaccard index, Kendall’s τ, nDCG, percent of products with inconsistent prices, and average price difference, comparing treatments against controls to separate noise from personalization.
Limitations
- The paper states the study is limited to US e-commerce sites, US IP addresses, English searches, and US users; it can identify only positive instances of personalization, not absence, and leaves other features such as geolocation, HTTP Referer, browsing history, and purchase history to future work. It also notes unexplained personalization on Newegg and Sears and that experiments were conducted in spring 2014, so results may not hold in the future.
- The paper states that Expedia and Hotels.com could not be analyzed from the AMT experiment because A/B testing made comparison and control results rarely match; rental car steering was not examined because travel sites present cars in a deterministic grid; and the authors could not infer which specific AMT user characteristics triggered personalization.
- The text makes evident that Home Depot’s differences may be artifacts of different server-side implementations for desktop and mobile rather than an explicit personalization algorithm, though they still qualify under the paper’s definition; synthetic purchase-history treatments may not embody realistic user behavior but were designed as unambiguous signals.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 12 Not checked · 107 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We develop a measurement infrastructure that is able to distinguish genuine personalization of e-commerce sites from other sources of noise; this methodology is based on previous work on measuring personalization of web search services [17].”
What the sources say
- Supportedcites [17] [17] A. Hannak, P. Sapiezy´ nski, A. M. Kakhki, B. Krishnamurthy, D. Lazer, A. Mislove, and C. Wilson. Measuring Personalization of Web Search. WWW , 2013. - source
The citation supports the methodological precedent, not the citing authors’ claim that they developed an e-commerce measurement infrastructure. The source’s Concluding Discussion (§7) describes a methodology for measuring web-search personalization that controls for sources of noise; its Introduction (§1) says the methodology can generalize to product search. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find evidence for some form of personalization on nine of these e-commerce sites.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To summarize our findings in this section: we find evidence for price steering and price discrimination on four general retailers and five travel sites.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, these four sites do exhibit significant noise: Kendall's τ is ≤ 0.83 in all four cases.”
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, we find no correlation between browser/OS choice and high personalization.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
66 more claims
Create a free account to see all 126 claims and to verify the 5 references still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 35
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, it is difficult to discover what the actual reason for a price fluctuation is and enforce accordingly -if a price fluctuation is even observed at all (Mikians et al., 2012;Hannak et. al., 2014).”
From Discrimination grounds and personalised pricing: Consumer perceptions of fairness, norm alignment, legality, and trust in markets (2024) · cites this paper as Hannak et. al., 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Social data are being leveraged to make inferences about how much to pay for a product (Hannak et al., 2014), about the likelihood of being a terrorist or about users health (Yom-Tov, 2016) and employability (Rosenblat et al., 2014).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Hannak et al., 2014
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Whenever the confidence intervals around these fractions for two audiences are non-overlapping, we can make a claim that the gender or racial makeups of two audiences are significantly different [17].”
From Discrimination through Optimization (2019) · cites this paper as [17]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, the converse is not true: overlapping confidence intervals do not necessarily mean that the means are not different (see Figure 4 in [17] for explanation).”
From Discrimination through Optimization (2019) · cites this paper as [17]
Experimental — accuracy not yet measuredIn the citing paper
“Similar audits have turned up issues of bias reinforcement in systems that guide police to portions of a city to patrol [36], and in e-commerce platforms [35].”
From The fallacy of inscrutability (2018) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1. INTRODUCTION · p. 2
“Our real-world data indicates that eight of these sites implement personalization”
§ 4.5 Summary · p. 7
“we find evidence for price steering and price discrimination on four general retailers and five travel sites.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“prices fall by ≈ $15 on ≈ 5% of hotels (or 5 out 50 per page) for iOS users.”
Reported 5 · recomputed 10
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2 Implementation · p. 4
“| Staples | http://sears.com | Office supplies |”
§ 3.2 Implementation · p. 4
“| Sears | http://sears.com | Clothes, housewares |”
Citations
Cites
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
A NEW MEASURE OF RANK CORRELATION
KENDALL, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,269
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
(2011) · Cited by 4,430
Information Rules: A Strategic Guide to the Network Economy
Shaw, M. J., Shapiro, C., & Varian, H. R. (2000) · The Academy of Management Review · doi:10.2307/259025 · Cited by 4,124
Becker, W. E., Shapiro, C., & Varian, H. R. (1999) · The Journal of Economic Education · doi:10.1080/00220489909595956 · Cited by 2,685
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Information Rules: A Strategic Guide to the Network Economy
Becker, W. E., Shapiro, C., & Varian, H. R. (1999) · The Journal of Economic Education · doi:10.2307/1183273 · Cited by 2,557
The proof and measurement of association between two things
Spearman, C. (2010) · International Journal of Epidemiology · doi:10.1093/ije/dyq191 · Cited by 1,681
Google news personalization: scalable online collaborative filtering
Das, A. S., Datar, M., Garg, A., & Rajaram, S. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242610 · Cited by 1,511
Research works printed but not linked: 1
- T. Wadhwa . How Advertisers Can Use Your Personal Information to Make You Pay Higher Prices. Huffington Post , 2014 . http://www.huffingtonpost.com/tarun-wadhwa/how-advertisers-can-use-y_b_4703013.html. T. Wadhwa. How Advertisers Can Use Your Personal Information to Make You Pay Higher Prices. Huffington Post, 2014. http://www.huffingtonpost.com/tarun-wadhwa/how-advertisers-can-use-y_b_4703013.html.
Cited by
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“Social data are being leveraged to make inferences about how much to pay for a product (Hannak et al., 2014), about the likelihood of being a terrorist or about users health (Yom-Tov, 2016) and employability (Rosenblat et al., 2014).”
BackgroundOnline Tracking: A 1-million-site Measurement and Analysis
Englehardt, S., & Narayanan, A. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978313 · Cited by 647
“Many other practices that raise privacy or ethical concerns have been studied: price discrimination, where a site shows different prices to different consumers for the same product [19, 63]; steering, a gentler form of price discrimination where a product search shows differently-priced results for different users [32…”
BackgroundAuditing radicalization pathways on YouTube
Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372879 · Cited by 580
“Previous large scale algorithmic auditing include measuring discrimination on AirBnB [14], personalization onweb search [19] and price discrimination on e-commerce web sites [20].”
BackgroundMathur, A., Kshirsagar, M., & Mayer, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445610 · Cited by 331
“know why they are shown a limited set of choices [23].”
Background
Metadata
- Authors
- Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C.
- Affiliations
- Northeastern University, Boston, MA, USA
- Northeastern University (US)
- Venue
- Proceedings of the 2014 Conference on Internet Measurement Conference
- Publisher
- ACM
- Published
- 2014-11-05
- Pages
- 305-318
- OpenAlex
- W2161392727
- Cited by
- 405
- References
- 51
- Citations by year
- 2026: 30 · 2025: 57 · 2024: 25 · 2023: 46 · 2022: 36 · 2021: 41 · 2020: 28 · 2019: 34 · 2018: 40 · 2017: 28 · 2016: 25 · 2015: 14 · 2014: 1
- FWCI
- 63.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Internet Measurement Conference (IMC) (publishedVersion)
- Funders
- National Science Foundation (CHS- 1408345)
- Division of Computer and Network Systems (CNS-1054233)
- Amazon Web Services in Education
- Army Research Office (W911NF-12-1-0556)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- price discrimination, e-commerce personalization, recommender systems, personalized pricing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring Price Discrimination and Steering on E-commerce Web Sites. https://app.cochat.ai/papers/10.1145/2663716.2663744
- The paper
- Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014). Measuring Price Discrimination and Steering on E-commerce Web Sites. Proceedings of the 2014 Conference on Internet Measurement Conference. https://doi.org/10.1145/2663716.2663744
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.