CoChat
Sign Up
Cited by 312Advanced Bandit Algorithms ResearchOptimal Experimental Design MethodsAdvanced Causal Inference Techniques

The DOI matches its Crossref record; 228 of the 235 references it cites were found in a registry or an index.

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Kohavi, Ron, Tang, Diane, Xu, Ya·2020·Cambridge University Press eBooks
DOI 10.1017/9781108653985
Open this paper in CoChat — free account

Abstract

Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments • Define key metrics and ideally an Overall Evaluation Criterion • Test for trustworthiness of the results and alert experimenters to violated assumptions • Build a scalable platform that lowers the marginal cost of experiments close to zero • Avoid pitfalls like carryover effects and Twyman's law • Understand how statistical issues play out in practice.

Overview

Integrity signals

Registry

Crossref · 10.1017/9781108653985

Cites

228 of 228 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 235 references

Research works printed but not linked: 46

  1. 10.2307/2331554 · DOI · Paper page
  2. 10.1017/S1062798700002660 · DOI · Paper page
  3. Xia, 2019, ICSE: 41st ACM/IEEE International Conference on Software Engineering
  4. Owen, Art , and Hal Varian . 2009. Oxford Centre for Evidence-based Medicine – Levels of Evidence. March. www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/.
  5. Schurman, Eric , and Jake Brutlag . 2009. “Performance Related Changes and their User Impact.” Velocity 09: Velocity Web Performance and Operations Conference. www.youtube.com/watch?v=bQSE51-gr2s and www.slideshare.net/dyninc/the-user-and-business-impact-of-server-delays-additional-bytes-and-http-chunking-in-web-search-presentation.
  6. Mudholkar, Govind S. , and E. Olusegun George . 1979. “The Logit Method for Combining Probablilities.” Edited by J. Rustagi . Symposium on Optimizing Methods in Statistics.” Academic Press. 345–366. https://apps.dtic.mil/dtic/tr/fulltext/u2/a049993.pdf.
  7. Mosteller, 1983, Clinical Trials
  8. McFarland, 2012, Experiment!: Website Conversion Rate Optimization with A/B and Multivariate Testing
  9. 1955, A Million Random Digits with 100,000 Normal Deviates
  10. Fabijan, 2018, Euromicro Conference on Software Engineering and Advanced Applications (SEAA)
  11. Kohavi, Ron . 2014. “Customer Review of A/B Testing: The Most Powerful Way to Turn Clicks Into Customers.” Amazon.com. May 27. www.amazon.com/gp/customer-reviews/R44BH2HO30T18.
  12. Xu, 2018, 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 895
  13. Dickson, 1999, The Official Rules and Explanations: The Original Guide to Surviving the Electronic Age With Wit, Wisdom, and Laughter
  14. Deng, 2017, WSDM: The Tenth International Conference on Web Search and Data Mining
  15. Lee, 2018, KDD 2018: The 24th ACM Conference on Knowledge Discovery and Data Mining
  16. McGue, 2014, Introduction to Human Behavioral Genetics, Unit 2: Twins: A Natural Experiment
  17. Kohavi, Ron . 2016. “Pitfalls in Online Controlled Experiments.” CODE ’16: Conference on Digital Experimentation. MIT. https://bit.ly/Code2016Kohavi.
  18. Saint-Jacques, 2018, Workshop on Information Systems and Exonomics
  19. Fabijan, 2019, KDD ‘19: The 25th SIGKDD International Conference on Knowledge Discovery and Data Mining
  20. Goward, 2012, You Should Test That: Conversion Optimization for More Leads, Sales and Profit or The Art and Science of Optimized Marketing
  21. 250 Deng, Shaojie , Roger Longbotham , Toby Walker , and Ya Xu . 2011. “Choice of Randomization Unit in Online Controlled Experiments.” Joint Statistical Meetings Proceedings. 4866–4877.
  22. Gelman, 1997, Poststratification into Many Categories Using Hierarchical Logistic Regression, Survey Methdology, 23, 127
  23. Groves, 2009, Survey Methodology
  24. Charles, 2004, Handbook of Practical Program Evaluation
  25. Razali, 2011, Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lillefors and Anderson-Darling tests, Journal of Statistical Modeling and Analytics, January, 1, 21
  26. Chatham, 2004, A Primer on A/B Testing
  27. HIPAA Journal . 2018. What is Considered Protected Health Information Under HIPAA. April 2. https://www.hipaajournal.com/what-is-considered-protected-health-information-under-hipaa/.
  28. Pekelis, 2005, Web Site Measurement Hacks
  29. Goldratt, 1990, The Haystack Syndrome
  30. Keppel, 1992, Introduction to Design and Analysis
  31. Li, 2010, WWW 2010: Proceedings of the 19th International Conference on World Wide Web
  32. McChesney, 2012, The 4 Disciplines of Execution: Achieving Your Wildly Important Goals
  33. Lu, 2014, KDD User Engagement Optimization Workshop
  34. Peterson, 2004, Web Analytics Demystified: A Marketer’s Guide to Understanding How Your Web Site Affects Your Business
  35. 10.1023/B:AIRE.0000045502.10941.a9 · DOI · Paper page
  36. Hohnhold, 2015, Proceedings 21st Conference on Knowledge Discovery and Data Mining (KDD 2015)
  37. Fabijan, 2017, ICSE ’17 Proceedings of the 39th International Conference on Software Engineering, 770
  38. Kohavi, Ron . 2010. “Online Controlled Experiments: Listening to the Customers, not to the HiPPO.” Keynote at EC10: the 11th ACM Conference on Electronic Commerce. www.exp-platform.com/Documents/2010-06%20EC10.pptx.
  39. Kohavi, 2003, Real-world Insights from Mining Retail E-Commerce Data
  40. Meenan, Patrick , Chao (Ray) Feng , and Mike Petrovich . 2013. “Going Beyond Onload – How Fast Does It Feel?” Velocity: Web Performance and Operations conference, October 14–16. http://velocityconf.com/velocityny2013/public/schedule/detail/31344.
  41. Liu, 2018, Joint Statistical Meeting, Statistical Consulting Section, 2849
  42. Brutlag, Jake , Zoe Abrams , and Pat Meenan . 2011. “Above the Fold Time: Measuring Web Page Performance Visually.” Velocity: Web Performance and Operations Conference.
  43. Lewis, Randall A. , Justin M. Rao , and David Reiley . 2011. “Proceedings of the 20th ACM International World Wide Web Conference (WWW20).” 157–166. https://ssrn.com/abstract=2080235.
  44. Loukides, 2018, Ethics and Data Science
  45. Vaver, 2012, Periodic Measuement of Advertising Effectiveness Using Multiple-Test Period Geo Experiments
  46. ACM. 2018. ACM Code of Ethics and Professional Conduct. June 22. www.acm.org/code-of-ethics.

Cited by

Sorted by citations

See all 312

Related

Metadata

Authors
Kohavi, R., Tang, D., & Xu, Y.
Affiliations
  • Microsoft (United States) (US)
  • Google (United States) (US)
  • LinkedIn (United States) (US)
ORCID
Kohavi, Ron
Venue
Cambridge University Press eBooks
Publisher
Cambridge University Press
Published
2020-03-13
DOI
10.1017/9781108653985
OpenAlex
W3011423189
Cited by
297
References
235
Citations by year
2026: 64 · 2025: 53 · 2024: 46 · 2023: 39 · 2022: 34 · 2021: 19 · 2020: 38
FWCI
16.61
Type
book
Language
en
Open access
closed
Licence
www.cambridge.org/core/terms (unspecified)
Locations
Cambridge University Press eBooks (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
A/B testing, experimentation, data-driven decision making, hypothesis testing, statistical significance, residual effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review