The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Open accessAbstract
To partly address people’s concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google’s ads, and Ad Settings interact. AdFisher can run browser-based experiments and analyze data using machine learning and significance tests. Our tool uses a rigorous experimental design and statistical analysis to ensure the statistical soundness of our results. We use AdFisher to find that the Ad Settings was opaque about some features of a user’s profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse changed the ads shown but not the settings page. We also found that setting the gender to female resulted in getting fewer instances of an ad related to high paying jobs than setting it to male. We cannot determine who caused these findings due to our limited visibility into the ad ecosystem, which includes Google, advertisers, websites, and users. Nevertheless, these results can form the starting point for deeper investigations by either the companies themselves or by regulatory bodies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, it has been shown that setting the gender to ‘female’ on Google’s AdSetting’s page results in fewer advertisements for high-paying jobs (Datta et al., 2015).”
From Algorithmic profiling as a source of hermeneutical injustice (2025) · cites this paper as Datta et al., 2015
In the citing paper
“Existing audits for fairness evaluate models by quantifying disparities in performance across groups ([16], [27], [70]), which can be seen as additional metrics to prevent against biased evaluation.”
From Fairness and explainability in automatic decision-making systems. A challenge for computer science and law (2023) · cites this paper as [27]
In the citing paper
“However, not all automated accounts are malicious. Some of them are used to post important updates about weather or other topics, such as emergency alerts. Others are designed by third party researchers to understand and audit system behavior”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Datta et al., 2015
In the citing paper
““Google showed the simulated males ads from a certain career coaching agency that promised large salaries more frequently than the simulated females, a finding suggestive of discrimination.””
From Discrimination, artificial intelligence, and algorithmic decision-making (2025) · cites this paper as Datta, Tschantz and Datta 2015
In the citing paper
“Researchers note that it is unclear why women were shown fewer ads for high-paying jobs, because of the opaqueness of the system”
From Discrimination, artificial intelligence, and algorithmic decision-making (2025) · cites this paper as Datta, Tschantz and Datta 2015
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
Clopper, C. J., & Pearson, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,404
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Encyclopedia of Measurement and Statistics
Salkind, N. (2007) · Sage eBooks · doi:10.4135/9781412952644 · Cited by 1,420
(2013) · Cited by 1,126
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · cited 1×
“Source code for AdFisher, raw data, and additional details about our experiments can be found at http://www.cs.cmu.edu/ ∼ mtschant/ife/”
Permutation, Parametric and Bootstrap Tests of Hypotheses
(2005) · Springer Series in Statistics · doi:10.1007/b138696 · Cited by 707
Identifiability, Exchangeability, and Epidemiological Confounding
GREENLAND, S., & ROBINS, J. M. (1986) · International Journal of Epidemiology · doi:10.1093/ije/15.3.413 · Cited by 686
Cited by
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196
“Some solutions perhaps wisely abandon answering the ‘why’ question and devise metrics that can, in other ways, evaluate discrimination (i.e. Datta et al., 2015).”
BackgroundAnanny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016) · arXiv · doi:10.48550/arxiv.1607.06520 · Cited by 1,615
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
Metadata
- Authors
- Datta, A., Tschantz, M. C., & Datta, A.
- Affiliations
- Carnegie Mellon University (US)
- International Computer Science Institute (US)
- Venue
- Proceedings on Privacy Enhancing Technologies
- Published
- 2015-04-01
- Volume
- 2015
- Issue
- 1
- Pages
- 92-112
- ISSN
- 2299-0984 (ISSN-L)
- OpenAlex
- W1648303880
- Cited by
- 676
- References
- 22
- Citations by year
- 2026: 13 · 2025: 33 · 2024: 68 · 2023: 96 · 2022: 100 · 2021: 87 · 2020: 80 · 2019: 75 · 2018: 61 · 2017: 28 · 2016: 26 · 2015: 9
- FWCI
- 137.74
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings on Privacy Enhancing Technologies (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- online tracking, user profiles, algorithmic discrimination, online advertising, personalized advertising
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. https://app.cochat.ai/papers/10.1515/popets-2015-0007
- The paper
- Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proceedings on Privacy Enhancing Technologies. https://doi.org/10.1515/popets-2015-0007
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.