Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Measuring the causal effects of online advertising (adfx) on user behavior is important to the health of the WWW publishing industry. In this paper, using three controlled experiments, we show that observational data frequently lead to incorrect estimates of adfx. The reason, which we label "activity bias," comes from the surprising amount of time-based correlation between the myriad activities that users undertake online. In Experiment 1, users who are exposed to an ad on a given day are much more likely to engage in brand-relevant search queries as compared to their recent history for reasons that had nothing do with the advertisement. In Experiment 2, we show that activity bias occurs for page views across diverse websites. In Experiment 3, we track account sign-ups at a competitor's (of the advertiser) website and find that many more people sign-up on the day they saw an advertisement than on other days, but that the true "competitive effect" was minimal. In all three experiments, exposure to a campaign signals doing "more of everything" in given period of time, making it difficult to find a suitable "matched control" using prior behavior. In such cases, the "match" is fundamentally different from the exposed group, and we show how and why observational methods lead to a massive overestimate of adfx in such circumstances.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Doubly robust estimation in missing data and causal inference models
Bang, H., & Robins, J. M. (2005) · Biometrics · doi:10.1111/j.1541-0420.2005.00377.x · Cited by 1,998
Cited by
Inferring causal impact using Bayesian structural time-series models
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015) · The Annals of Applied Statistics · doi:10.1214/14-aoas788 · Cited by 961
When Does Retargeting Work? Information Specificity in Online Advertising
Lambrecht, A., & Tucker, C. (2013) · Journal of Marketing Research · doi:10.1509/jmr.11.0503 · Cited by 508
Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment
Blake, T., Nosko, C., & Tadelis, S. (2015) · Econometrica · doi:10.3982/ecta12423 · Cited by 354
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019) · Marketing Science · doi:10.1287/mksc.2018.1135 · Cited by 287
“…a variety of matching and regression-based methods and compare their results with those obtained from the RCTs. Earlier work to evaluate such observational models had limited individual-level data and considered a narrow set of models (Lewis, Rao, and Reiley 2011, Blake, Nosko, and Tadelis 2015).”
InfluentialConsumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment
Blake, T., Nosko, C., & Tadelis, S. (2014) · National Bureau of Economic Research · doi:10.3386/w20171 · Cited by 217
Lewis, R. A., & Reiley, D. H. (2014) · Quantitative Marketing and Economics · doi:10.1007/s11129-014-9146-6 · Cited by 211
Self-citationMapping the customer journey: Lessons learned from graph-based online attribution modeling
Anderl, E., Becker, I., von Wangenheim, F., & Schumann, J. H. (2016) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2016.03.001 · Cited by 186
How do firms make money selling digital goods online?
Lambrecht, A., Goldfarb, A., Bonatti, A., Ghose, A., Goldstein, D. G., Lewis, R., Rao, A., Sahni, N., & Yao, S. (2014) · Marketing Letters · doi:10.1007/s11002-014-9310-5 · Cited by 167
“This “activity bias” (Lewis et al. 2011) led the advertiser to believe that their advertising was significantly more effective than the researchers’ experiments indicate.”
BackgroundSelf-citationBig Data and the danger of being precisely inaccurate
McFarland, D. A., & McFarland, H. R. (2015) · Big Data & Society · doi:10.1177/2053951715602495 · Cited by 81
Metadata
- Authors
- Lewis, R. A., Rao, J. M., & Reiley, D. H.
- Venue
- Proceedings of the 20th international conference on World wide web
- Publisher
- ACM
- Published
- 2011-03-28
- Pages
- 157-166
- OpenAlex
- W3126019886
- Cited by
- 105
- References
- 1
- FWCI
- 31.47
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.