Challenges in measuring online advertising systems
Abstract
Online advertising supports many Internet services, such as search, email, and social networks. At the same time, there are widespread concerns about the privacy loss associated with user targeting. Yet, very little is publicly known about how ad networks operate, especially with regard to how they use user information to target users. This paper takes a first principled look at measurement methodologies for ad networks. It proposes new metrics that are robust to the high levels of noise inherent in ad distribution, identifies measurement pitfalls and artifacts, and provides mitigation strategies. It also presents an analysis of how three different classes of advertising -- search, contextual, and social networks, use user profile information today.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Static DNS entries direct all query traffic to specific IP addresses of the retailers to eliminate errors arising from differences between datacenters.
“Second , we use static DNS entries to direct all of our query traffic to specific IP addresses of the retailers. This eliminates errors arising from differences between datacenters.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [16]
SSH tunnels forward the traffic of each treatment to a unique IP address in a /24 subnet to ensure that any effects due to IP geolocation affect all results equally.
“Third , although all PhantomJS instances execute on one machine, we use SSH tunnels to forward the traffic of each treatment to a unique IP address in a /24 subnet. This process ensures that any effects due to IP geolocation will affect all results equally.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [16]
All searches for a given query are executed at the same time to eliminate differences in results due to temporal effects.
“first , all searches for a given query are executed at the same time. This eliminates differences in results due to temporal effects.”
From Measuring price discrimination and steering on e-commerce web sites (2014) · cites this paper as [16]
Cited by
Sorted by citations
Algorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · see citing works
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Datta, A., Tschantz, M. C., & Datta, A. (2015) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2015-0007 · Cited by 676 · see citing works
Bias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649 · see citing works
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 368 · see citing works
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · see citing works
Detecting price and search discrimination on the internet
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2012) · Proceedings of the 11th ACM Workshop on Hot Topics in Networks · doi:10.1145/2390231.2390245 · Cited by 244 · see citing works
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
Understanding what they do with what they know
Wills, C. E., & Tatar, C. (2012) · Proceedings of the 2012 ACM workshop on Privacy in the electronic society · doi:10.1145/2381966.2381969 · Cited by 79 · see citing works
Automated Data Quality Control System in Health and Demographic Surveillance System
Tlouyamma, J., & Mokwena, S. (2024) · Science, Engineering and Technology · doi:10.54327/set2024/v4.i2.125 · Cited by 1 · see citing works
- Authors
- Guha, S., Cheng, B., & Francis, P.
- Affiliations
- Microsoft Research (India) (IN)
- Max Planck Institute for Software Systems (DE)
- Venue
- Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
- Publisher
- ACM
- Published
- 2010-11
- Pages
- 81-87
- OpenAlex
- W2144093215
- Cited by
- 169
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 3 · 2022: 6 · 2021: 4 · 2020: 12 · 2019: 16 · 2018: 11 · 2017: 19 · 2016: 21 · 2015: 18 · 2014: 19 · 2013: 12 · 2012: 12
- FWCI
- 30.85
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Internet Measurement Conference (IMC) (publishedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Social Sciences › Sociology and Political Science › Digital Marketing and Social Media
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- online advertising, ad networks, measurement methods, user targeting, privacy, user profiling, search engine advertising, contextual advertising, social media advertising, measurement noise, measurement artifacts
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
