Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Abstract
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments • Define key metrics and ideally an Overall Evaluation Criterion • Test for trustworthiness of the results and alert experimenters to violated assumptions • Build a scalable platform that lowers the marginal cost of experiments close to zero • Avoid pitfalls like carryover effects and Twyman's law • Understand how statistical issues play out in practice.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Residents examined and gave iterative feedback on two different versions of the EHR tool using A/B testing.
“Residents examined and gave iterative feedback on two different versions of the EHR tool using A/B testing (a standardized comparison between two versions of a product, commonly used in software design, with different versions presented first or second randomly) (12) as part of a Microsoft PowerPoint slide deck and a virtual whiteboard.”
From Cocreating the ICU-PAUSE Tool for Intensive Care Unit–Ward Transitions (2022) · cites this paper as [12]
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170 · see citing works
Competitive Strategy: Techniques for Analyzing Industries and Competitors
(1980) · Journal of Sustainable Competitive Intelligence · doi:10.24883/iberoamericanic.v2i2.38 · Cited by 16,592 · see citing works
(2002) · Cited by 13,211 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883 · see citing works
Wasserman, S. S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,347 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467 · see citing works
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · see citing works
The balanced scorecard: Translating strategy into action
Norton, D. P. (1997) · Long Range Planning · doi:10.1016/s0024-6301(97)80925-9 · Cited by 8,244 · see citing works
The Structure of Scientific Revolutions
Kuhn, T. S. (1996) · doi:10.7208/chicago/9780226458106.001.0001 · Cited by 7,281 · see citing works
PE, & Billingsley, P. (1996) · Journal of the American Statistical Association · doi:10.2307/2291440 · Cited by 6,639 · see citing works
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · see citing works
Econometric policy evaluation: A critique
Lucas, R. E. (1976) · Carnegie-Rochester Conference Series on Public Policy · doi:10.1016/s0167-2231(76)80003-6 · Cited by 6,110 · see citing works
Statistical Methods for Meta-Analysis
Rosenbaum, P. R., Hedges, L., & Olkin, I. (1987) · Journal of the American Statistical Association · doi:10.2307/2289186 · Cited by 6,028 · see citing works
Casella, G., & Berger, R. L. (1993) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003456285 · Cited by 5,835 · see citing works
Porter, M. E. (2000) · Cited by 5,799 · see citing works
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557 · see citing works
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210 · see citing works
Experimental evidence of massive-scale emotional contagion through social networks
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320040111 · Cited by 3,687 · see citing works
Manipulation of the Running Variable in the Regression Discontinuity Design: A Density Test,
McCrary, J. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.005 · Cited by 3,600 · see citing works
A Survey of Outlier Detection Methodologies
Hodge, V., & Austin, J. (2004) · Artificial Intelligence Review · doi:10.1007/s10462-004-4304-y · Cited by 3,596 · see citing works
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · see citing works
Ries, E. (2011) · Cited by 3,330 · see citing works
An Introduction to the Bootstrap.
Buckland, S. T., Efron, B., & Tibshirani, R. J. (1994) · Biometrics · doi:10.2307/2532810 · Cited by 3,236 · see citing works
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
Econometric policy evaluation: A critique
Pritchett, D. V. (1976) · Carnegie-Rochester Conference Series on Public Policy · doi:10.1016/s0167-2231(76)80006-1 · Cited by 2,961 · see citing works
Evaluation: A Systematic Approach
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (1979) · Cited by 2,943 · see citing works
Estimation and Inference of Heterogeneous Treatment Effects using Random Forests
Wager, S., & Athey, S. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1319839 · Cited by 2,917 · see citing works
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772 · see citing works
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562 · see citing works
Phillips, A. W. (1958) · Economica · doi:10.1111/j.1468-0335.1958.tb00003.x · Cited by 2,390 · see citing works
Equipoise and the Ethics of Clinical Research
Freedman, B. (1987) · New England Journal of Medicine · doi:10.1056/nejm198707163170304 · Cited by 2,339 · see citing works
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301 · see citing works
Card, D., & Krueger, A. B. (2000) · American Economic Review · doi:10.1257/aer.90.5.1397 · Cited by 2,208 · see citing works
Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania
Card, D., & Krueger, A. B. (1993) · National Bureau of Economic Research · doi:10.3386/w4509 · Cited by 2,072 · see citing works
Evaluation: A Systematic Approach
Wallace, D. P. (2005) · Library & Information Science Research · doi:10.1016/j.lisr.2004.09.009 · Cited by 1,938 · see citing works
Why Propensity Scores Should Not Be Used for Matching
King, G., & Nielsen, R. (2019) · Political Analysis · doi:10.1017/pan.2019.11 · Cited by 1,706 · see citing works
Statistics for experimenters: design, innovation, and discovery
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3429 · Cited by 1,657 · see citing works
On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9
Neyman, J., Dabrowska, D. M., & Speed, T. P. (1990) · Statistical Science · doi:10.1214/ss/1177012031 · Cited by 1,616 · see citing works
Recursive partitioning for heterogeneous causal effects
Athey, S., & Imbens, G. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510489113 · Cited by 1,586 · see citing works
Handbook of Practical Program Evaluation
Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2015) · Wiley eBooks · doi:10.1002/9781119171386 · Cited by 1,471 · see citing works
Regression-discontinuity analysis: An alternative to the ex post facto experiment.
Thistlethwaite, D. L., & Campbell, D. T. (1960) · Journal of Educational Psychology · doi:10.1037/h0044319 · Cited by 1,469 · see citing works
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · see citing works
Phillips, A. W. (1958) · Economica · doi:10.2307/2550759 · Cited by 1,443 · see citing works
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408 · see citing works
Student (1908) · Biometrika · doi:10.1093/biomet/6.1.1 · Cited by 1,363 · see citing works
Samarati, P., & Sweeney, L. (2018) · doi:10.1184/r1/6625469.v1 · Cited by 1,344 · see citing works
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · see citing works
Samarati, P., & Sweeney, L. (2018) · doi:10.1184/r1/6625469 · Cited by 1,341 · see citing works
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322 · see citing works
Statistical Methods for Research Workers
Fisher, B. R. A. (1926) · Journal of Comparative Pathology and Therapeutics · doi:10.1016/s0368-1742(26)80018-9 · Cited by 1,287 · see citing works
The Knowing‐Doing Gap: How Smart Companies Turn Knowledge into Action
Pfeffer, J. (2001) · Supply Chain Management An International Journal · doi:10.1108/scm.2001.6.3.142.1 · Cited by 1,195 · see citing works
All of Statistics: A Concise Course in Statistical Inference
Wasserman, L. (2004) · Springer Texts in Statistics · doi:10.1007/978-0-387-21736-9 · Cited by 1,090 · see citing works
(1958) · Medical Entomology and Zoology · Cited by 1,059 · see citing works
Statistical design and analysis of experiments: with applications to engineering and science
(1990) · Choice Reviews Online · doi:10.5860/choice.27-2741 · Cited by 952 · see citing works
Evaluation : methods for studying programs and policies
Weiss, C. H. (1997) · Cited by 931 · see citing works
Encyclopedia of Machine Learning and Data Mining
Webb, G. I., & Sammut, C. (2017) · Springer eBooks · doi:10.1007/978-1-4899-7687-1 · Cited by 926 · see citing works
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 867 · see citing works
Users' Guides to the Medical Literature: IX. A Method for Grading Health Care Recommendations
Guyatt, G. H. (1995) · JAMA · doi:10.1001/jama.1995.03530220066035 · Cited by 846 · see citing works
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · Thammasat University Digital Collections · Cited by 838 · see citing works
Design of Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement
Nostrand, R. C. V. (2002) · Technometrics · doi:10.1198/004017002320256440 · Cited by 834 · see citing works
Key Performance Indicators: Developing, Implementing, and Using Winning KPIs
Parmenter, D. A. (2007) · Cited by 821 · see citing works
Statistics for Experimenters: Design, Innovation, and Discovery
Speed, T. P. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s139 · Cited by 791 · see citing works
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772 · see citing works
Testing Statistical Hypothesis.
Crouse, C. F., & Lehmann, E. L. (1960) · Biometrika · doi:10.2307/2332982 · Cited by 753 · see citing works
How to read a paper: The basics of evidence-based medicine
(1998) · Air Medical Journal · doi:10.1016/s1067-991x(98)90091-8 · Cited by 720 · see citing works
Controlled experiments on the web: survey and practical guide
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2008) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0114-1 · Cited by 691 · see citing works
Self-citationDesign of Experiments Using The Taguchi Approach: 16 Steps to Product and Process Improvement
Roy, R. (2001) · Cited by 678 · see citing works
All Of Statistic A Concise Course In Statistical Inference
Waserman, L. (2003) · Cited by 652 · see citing works
The Four Steps to the Epiphany: Successful Strategies for Products that Win
Blank, S. G. (2013) · Cited by 588 · see citing works
How to measure anything: finding the value of "intangibles" in business
(2008) · Choice Reviews Online · doi:10.5860/choice.45-6882 · Cited by 561 · see citing works
Essential Scrum: A Practical Guide to the Most Popular Agile Process
Rubin, K. S. (2012) · Cited by 546 · see citing works
How Should We Estimate Public Opinion in The States?
Lax, J. R., & Phillips, J. H. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00360.x · Cited by 526 · see citing works
Clean Code: A Handbook of Agile Software Craftsmanship
(2009) · Kybernetes · doi:10.1108/03684920910973252 · Cited by 496 · see citing works
Formal mode of statistical inference for causal effects
Rubin, D. B. (1990) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(90)90077-8 · Cited by 483 · see citing works
Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls
Park, D. K., Gelman, A., & Bafumi, J. (2004) · Political Analysis · doi:10.1093/pan/mph024 · Cited by 480 · see citing works
400: A Method for Combining Non-Independent, One-Sided Tests of Significance
Brown, M. B. (1975) · Biometrics · doi:10.2307/2529826 · Cited by 473 · see citing works
van Belle, G. (2002) · Wiley-Interscience eBooks · Cited by 464 · see citing works
Online controlled experiments at large scale
Kohavi, R., Deng, A., Frasca, B., Walker, T., XU, Y. N., & Pöhlmann, N. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488217 · Cited by 441 · see citing works
Self-citationProblems of Monetary Management: The UK Experience
Goodhart, C. (1984) · doi:10.1007/978-1-349-17295-5_4 · Cited by 440 · see citing works
The balanced scorecard: translating strategy into action
(1997) · Choice Reviews Online · doi:10.5860/choice.34-2839 · Cited by 432 · see citing works
A modern Bayesian look at the multi‐armed bandit
Scott, S. L. (2010) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.874 · Cited by 421 · see citing works
Clean Code : A Handbook of Agile Software Craftsmanship
Martin, R. (2008) · Cited by 418 · see citing works
Parascandola, J., & Marks, H. M. (1999) · The American Historical Review · doi:10.2307/2650255 · Cited by 417 · see citing works
Mastering 'metrics: the path from cause to effect
(2015) · Choice Reviews Online · doi:10.5860/choice.189854 · Cited by 389 · see citing works
Experimentation Matters: Unlocking the Potential of New Technologies for Innovation
Thomke, S. H. (2003) · Medical Entomology and Zoology · Cited by 385 · see citing works
Parmenter, D. A. (2015) · Wiley eBooks · doi:10.1002/9781119019855 · Cited by 364 · see citing works
The New Palgrave Dictionary of Economics
Shipman, A., & Zezza, G. (2016) · Palgrave Macmillan UK eBooks · doi:10.1057/978-1-349-95121-5 · Cited by 357 · see citing works
Overlapping experiment infrastructure
Tang, D., Agarwal, A., O'Brien, D., & Meyer, M. (2010) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1835804.1835810 · Cited by 314 · see citing works
Self-citationThe Progress of Experiment: Science and Therapeutic Reform in the United States, 1900-1990
Crouch, R. A. (1998) · BMJ · doi:10.1136/bmj.316.7137.1101a · Cited by 288 · see citing works
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019) · Marketing Science · doi:10.1287/mksc.2018.1135 · Cited by 287 · see citing works
An Additive Method for Combining Probability Values from Independent Experiments
Edgington, E. S. (1972) · The Journal of Psychology · doi:10.1080/00223980.1972.9924813 · Cited by 280 · see citing works
Seven myths of randomisation in clinical trials
Senn, S. J. (2012) · Statistics in Medicine · doi:10.1002/sim.5713 · Cited by 277 · see citing works
‘Improving ratings’: audit in the British University system
Strathern, M. (1997) · European Review · doi:10.1017/s1062798700002660 · Cited by 275 · see citing works
How to read a paper : getting your bearings (deciding what the paper is about)
Greenhalgh, T. (1997) · BMJ · doi:10.1136/bmj.315.7102.243 · Cited by 272 · see citing works
Huang, J. J. S., White, R. W., & Dumais, S. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979125 · Cited by 270 · see citing works
Causal inference in economics and marketing
Varian, H. R. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510479113 · Cited by 256 · see citing works
Deming, Data and Observational Studies
Young, S. S., & Karr, A. F. (2011) · Significance · doi:10.1111/j.1740-9713.2011.00506.x · Cited by 235 · see citing works
Measuring the user experience on a large scale: user-centered metrics for web applications
Rodden, K., Hutchinson, H., & Fu, X. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753687 · Cited by 233 · see citing works
Large-scale validation and analysis of interleaved search evaluation
Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012) · ACM Transactions on Information Systems · doi:10.1145/2094072.2094078 · Cited by 228 · see citing works
Metrics: You are what you measure!
Hauser, J., & Katz, G. (1998) · European Management Journal · doi:10.1016/s0263-2373(98)00029-2 · Cited by 228 · see citing works
Seven rules of thumb for web site experimenters
Kohavi, R., Deng, A., Longbotham, R., & XU, Y. N. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623341 · Cited by 228 · see citing works
Self-citationImproving the sensitivity of online controlled experiments by utilizing pre-experiment data
Deng, A., XU, Y. N., Kohavi, R., & Walker, T. (2013) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2433396.2433413 · Cited by 224 · see citing works
Self-citationTrustworthy online controlled experiments
Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & XU, Y. N. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339653 · Cited by 223 · see citing works
Self-citationMadansky, A., Rossi, P. H., Wright, J. D., & Anderson, A. B. (1985) · Journal of the American Statistical Association · doi:10.2307/2288504 · Cited by 215 · see citing works
From Infrastructure to Culture
XU, Y. N., Chen, N., Fernandez, A., Sinno, O., & Bhasin, A. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2788602 · Cited by 207 · see citing works
Self-citationSimultaneous Statistical Inference
Dickhaus, T. (2014) · Springer eBooks · doi:10.1007/978-3-642-45182-9 · Cited by 195 · see citing works
Transforming performance measurement: rethinking the way we measure and drive organizational success
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5748 · Cited by 193 · see citing works
Controlled trials: the 1948 watershed
Doll, R. (1998) · BMJ · doi:10.1136/bmj.317.7167.1217 · Cited by 191 · see citing works
Mastering ‘Metrics: The Path from Cause to Effect
Gardebroek, C. (2015) · European Review of Agricultural Economics · doi:10.1093/erae/jbv011 · Cited by 190 · see citing works
Graph cluster randomization: network exposure to multiple universes
Ugander, J., Karrer, B., Backstrom, L., & Kleinberg, J. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487695 · Cited by 183 · see citing works
Designing and deploying online field experiments
Bakshy, E., Eckles, D., & Bernstein, M. S. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2567967 · Cited by 181 · see citing works
Routinely collected data and comparative effectiveness evidence: promises and limitations
Hemkens, L. G., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · Canadian Medical Association Journal · doi:10.1503/cmaj.150653 · Cited by 181 · see citing works
Uncontrolled: the surprising payoff of trial-and-error for business, politics, and society
(2012) · Choice Reviews Online · doi:10.5860/choice.50-2137 · Cited by 180 · see citing works
How to Read a Paper, the Basics of Evidence-Based Medicine:
Thompson, J. (2015) · Occupational Medicine · doi:10.1093/occmed/kqu187 · Cited by 177 · see citing works
Tyler, M. E., & Ledford, J. (2006) · Cited by 173 · see citing works
(1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90036-s · Cited by 161 · see citing works
Seven pitfalls to avoid when running controlled experiments on the web
Crook, T., Frasca, B., Kohavi, R., & Longbotham, R. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557139 · Cited by 157 · see citing works
Self-citationOnline Experimentation at Microsoft
Kohavi, R., Crook, T. R., & Longbotham, R. (2009) · Cited by 153 · see citing works
Self-citationA/B Testing: The Most Powerful Way to Turn Clicks Into Customers
Siroker, D., & Koomen, P. (2013) · Cited by 150 · see citing works
Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs
Doerr, J. (2018) · Cited by 143 · see citing works
Lean Analytics: Use Data to Build a Better Startup Faster
Croll, A., & Yoskovitz, B. (2013) · Cited by 140 · see citing works
Of Rats, Rice, and Race: The Great Hanoi Rat Massacre, an Episode in French Colonial History
Vann, M. G. (2003) · French Colonial History · doi:10.1353/fch.2003.0027 · Cited by 140 · see citing works
Top Challenges from the first Practical Online Controlled Experiments Summit
Gupta, S., Kohavi, R., Tang, D., XU, Y. N., Andersen, R., Bakshy, E., Cardin, N. J., Chandran, S., Chen, N., Coey, D., et al. (2019) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3331651.3331655 · Cited by 140 · see citing works
Self-citationFabijan, A., Dmitriev, P., Olsson, H. H., & Bosch, J. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.76 · Cited by 138 · see citing works
Chaplin, C. (1964) · Cited by 132 · see citing works
The Surprising Power of Online Experiments
Kohavi, R., & Thomke, S. H. (2017) · Cited by 131 · see citing works
Self-citationRamesh Johari, B., Koomen, P., Pekelis, L., & Walsh, D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3097992 · Cited by 124 · see citing works
Exploring Data: An Introduction to Data Analysis for Social Scientists
Marsh, C., & Elliott, J. (1988) · Cited by 123 · see citing works
Gui, H., XU, Y. N., Bhasin, A., & Han, J. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741081 · Cited by 121 · see citing works
Self-citationData-Informed Leadership in Education.
Knapp, M. S., Swinnerton, J. A., Copland, M. A., & Monpas-Huber, J. (2006) · Cited by 120 · see citing works
Selection bias and the perils of benchmarking.
Denrell, J. C. (2005) · PubMed · Cited by 116 · see citing works
Testing a Statistical Hypothesis
Spokoiny, V. G., & Dickhaus, T. (2014) · Springer texts in statistics · doi:10.1007/978-3-642-39909-1_6 · Cited by 109 · see citing works
Rethinking Timing of First Sex and Delinquency
Harden, K. P., Mendle, J., Hill, J. E., Turkheimer, E. N., & Emery, R. E. (2007) · Journal of Youth and Adolescence · doi:10.1007/s10964-007-9228-9 · Cited by 109 · see citing works
Evaluating online ad campaigns in a pipeline
Chan, D. X., Ge, R., Gershony, O., Hesterberg, T., & Lambert, D. (2010) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1835804.1835809 · Cited by 108 · see citing works
Optimized interleaving for online retrieval evaluation
Radlinski, F., & Craswell, N. (2013) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2433396.2433429 · Cited by 106 · see citing works
The Knowing‐Doing Gap: How Smart Companies Turn Knowledge into Action
(2000) · Supply Chain Management An International Journal · doi:10.1108/scm.2000.5.5.261.1 · Cited by 105 · see citing works
Burrell, Q. L. (2003) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.298_3.x · Cited by 102 · see citing works
Unexpected results in online controlled experiments
Kohavi, R., & Longbotham, R. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964905 · Cited by 100 · see citing works
Self-citationCustomer-centric pricing: The surprising secret for profitability
Cross, R. G., & Dixit, A. (2005) · Business Horizons · doi:10.1016/j.bushor.2005.04.005 · Cited by 99 · see citing works
Azevedo, E. M., Deng, A., Olea, J. L. M., Rao, J. M., & Weyl, E. G. (2020) · Journal of Political Economy · doi:10.1086/710607 · Cited by 99 · see citing works
The Cambridge Handbook of Consumer Privacy
(2018) · Cambridge University Press eBooks · doi:10.1017/9781316831960 · Cited by 97 · see citing works
Dmitriev, P., Gupta, S., Kim, D. W., & Vaz, G. J. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098024 · Cited by 96 · see citing works
Kyburg, H. E., & Teng, C. M. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511612947.009 · Cited by 95 · see citing works
Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies
Fabijan, A., Dmitriev, P., McFarland, C., Vermeer, L., Olsson, H. H., & Bosch, J. (2018) · Journal of Software Evolution and Process · doi:10.1002/smr.2113 · Cited by 94 · see citing works
Improving the Sensitivity of Online Controlled Experiments
Xie, H., & Aurisset, J. (2016) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939733 · Cited by 93 · see citing works
Bäckström, L., & Kleinberg, J. M. (2011) · International World Wide Web Conference (WWW) · doi:10.1145/1963405.1963492 · Cited by 90 · see citing works
The ethics of online research with unsuspecting users: From A/B testing to C/D experimentation
Benbunan‐Fich, R. (2016) · Research Ethics · doi:10.1177/1747016116680664 · Cited by 89 · see citing works
Grove, A. S. (2020) · Vahlen eBooks · doi:10.15358/9783800660469 · Cited by 88 · see citing works
Yes!: 50 Scientifically Proven Ways to Be Persuasive
Kohnen, J. (2009) · Quality Management Journal · doi:10.1080/10686967.2009.11918231 · Cited by 87 · see citing works
Objecting to experiments that compare two unobjectionable policies or treatments
Meyer, M. N., Heck, P. R., Holtzman, G. S., Anderson, S. M., Cai, W., Watts, D. J., & Chabris, C. F. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1820701116 · Cited by 87 · see citing works
High performance web sites : essential knowledge for frontend engineers
Souders, S. (2007) · CERN Document Server (European Organization for Nuclear Research) · Cited by 86 · see citing works
The Innovator's Hypothesis: How Cheap Experiments Are Worth More than Good Ideas
Schrage, M. (2014) · Cited by 86 · see citing works
The Effect of Providing Peer Information on Retirement Savings Decisions
Beshears, J. L., Choi, J., Laibson, D., Madrian, B. C., & Milkman, K. L. (2011) · National Bureau of Economic Research · doi:10.3386/w17345 · Cited by 85 · see citing works
Query logs alone are not enough
Grimes, C., Tang, D., & Russell, D. M. (2007) · Cited by 83 · see citing works
Self-citationTen Supplementary Analyses to Improve E-commerce Web Sites
Kohavi, R., & Parekh, R. G. (2003) · Cited by 81 · see citing works
Self-citationDmitriev, P., & Wu, X. W. (2016) · doi:10.1145/2983323.2983356 · Cited by 80 · see citing works
Pitfalls of long-term online controlled experiments
Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., & Vaz, G. J. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840744 · Cited by 79 · see citing works
Self-citationWeb Metrics: Proven Methods for Measuring Web Site Success
(2003) · Performance Measurement and Metrics · doi:10.1108/14678040310507905 · Cited by 77 · see citing works
Objective Bayesian Two Sample Hypothesis Testing for Online Controlled Experiments
Deng, A. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2740908.2742563 · Cited by 77 · see citing works
The Anatomy of a Large-Scale Experimentation Platform
Gupta, S., Ulanova, L., Bhardwaj, S., Dmitriev, P., Raff, P., & Fabijan, A. (2018) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2018.00009 · Cited by 77 · see citing works
Why marketplace experimentation is harder than it seems
Blake, T., & Coey, D. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602837 · Cited by 76 · see citing works
On Purpose and by Necessity: Compliance Under the GDPR
Basin, D., Debois, S., & Hildebrandt, T. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-662-58387-6_2 · Cited by 73 · see citing works
Web Metrics: Proven Methods for Measuring Web Site Success
Sterne, J. (2002) · Cited by 72 · see citing works
Applying the Delta Method in Metric Analytics
Deng, A., Knoblich, U., & Lu, J. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219919 · Cited by 72 · see citing works
Designing with Data: Improving the User Experience with A/B Testing
King, R., Churchill, E. F., & Tan, C. (2017) · Cited by 70 · see citing works
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3033144 · Cited by 68 · see citing works
Experimental Designs and Estimation for Online Display Advertising Attribution in Marketplaces
Barajas, J., Akella, R., Holtan, M., & Flores, A. (2016) · Marketing Science · doi:10.1287/mksc.2016.0982 · Cited by 67 · see citing works
Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
Deng, A., Lu, J., & Chen, S. (2016) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.33 · Cited by 67 · see citing works
Measuring Consumer Sensitivity to Audio Advertising: A Field Experiment on Pandora Internet RadioPreprint
Huang, J., Reiley, D., & Riabov, N. M. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3166676 · Cited by 66 · see citing works
Yes!: 50 Scientifically Proven Ways to Be Persuasive
Goldstein, N. J., Martin, S., & Cialdini, R. B. (2008) · Medical Entomology and Zoology · Cited by 65 · see citing works
Two Cheers for Corporate Experimentation: The A/B Illusion and the Virtues of Data-Driven Innovation
Meyer, M. N. (2015) · Colorado Law Scholarly Commons (University of Colorado Colorado Springs) · Cited by 63 · see citing works
On the accuracy of the Gaussian approximation
Tyurin, I. S. (2009) · Doklady Mathematics · doi:10.1134/s1064562409060155 · Cited by 63 · see citing works
What is a P-value anyway?: 34 stories to help you actually understand statistics
Vickers, A. J. (2010) · Choice Reviews Online · doi:10.5860/choice.48-0328 · Cited by 62 · see citing works
Online Experiments: Practical Lessons
Kohavi, R., Longbotham, R., & Walker, T. (2010) · Computer · doi:10.1109/mc.2010.264 · Cited by 62 · see citing works
Self-citationFramework and algorithms for network bucket testing
Katzir, L., Liberty, E., & Somekh, O. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187974 · Cited by 61 · see citing works
Malinas, G., & Bigelow, J. (2004) · Cited by 61 · see citing works
Evaluating Mobile Apps with A/B and Quasi A/B Tests
XU, Y. N., & Chen, N. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939703 · Cited by 61 · see citing works
Self-citationSuper returns to Super Bowl ads?
Stephens-Davidowitz, S. I., Varian, H. R., & Smith, M. D. (2017) · Quantitative Marketing and Economics · doi:10.1007/s11129-016-9179-0 · Cited by 61 · see citing works
Always Be Testing: The Complete Guide to Google Website Optimizer
Eisenberg, B., & Quarto-vonTivadar, J. (2008) · Cited by 59 · see citing works
Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments
Deng, A., & Hu, V. (2015) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2684822.2685307 · Cited by 57 · see citing works
Even Faster Web Sites: Performance Best Practices for Web Developers
Souders, S. (2009) · Cited by 56 · see citing works
Online Experimentation Diagnosis and Troubleshooting Beyond AA Validation
Zhao, Z., Chen, M., Matheson, D., & Stone, M. V. (2016) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.61 · Cited by 56 · see citing works
Insanely Simple: The Obsession That Drives Apple's Success
Segall, K. (2012) · Cited by 54 · see citing works
Do it wrong quickly: how the Web changes the old marketing rules
(2008) · Choice Reviews Online · doi:10.5860/choice.45-4478 · Cited by 52 · see citing works
Assigned tasks are not the same as self-chosen Web search tasks
Russell, D. M., & Grimes, C. (2007) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2007.91 · Cited by 52 · see citing works
Measuring Ad Effectiveness Using Geo Experiments
Vaver, J., & Koehler, J. (2011) · Cited by 52 · see citing works
Lean Customer Development: Building Products Your Customers Will Buy
Álvarez, C. (2014) · Cited by 52 · see citing works
Design and Analysis of Benchmarking Experiments for Distributed Internet Services
Bakshy, E., & Frachtenberg, E. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741082 · Cited by 51 · see citing works
Peterson, E. (2005) · CERN Document Server (European Organization for Nuclear Research) · Cited by 51 · see citing works
Clinical Trials-Issues and Approaches.
Healy, M. J. R., Shapiro, S. H., & Louis, T. (1984) · Biometrics · doi:10.2307/2530781 · Cited by 51 · see citing works
Edmonds, A., White, R. W., Morris, D., & Drucker, S. M. (2007) · Journal of Web Engineering · Cited by 49 · see citing works
Design and Analysis of Experiments in Networks: Reducing Bias from Interference
Eckles, D., Karrer, B., & Ugander, J. (2017) · Journal of Causal Inference · doi:10.1515/jci-2015-0021 · Cited by 49 · see citing works
At what degree of belief in a research hypothesis is a trial in humans justified?
Djulbegović, B., & Hozo, I. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00347.x · Cited by 48 · see citing works
Flagged first, then by citations
6 of the 235 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Wang, P. (2021) · MIS Quarterly · doi:10.25300/misq/2021/15864 · Cited by 265 · see citing works
A causal test of the strength of weak ties
Rajkumar, K., Saint-Jacques, G., Bojinov, I., Brynjolfsson, E., & Aral, S. (2022) · Science · doi:10.1126/science.abl4476 · Cited by 216 · see citing works
Fairness in Information Access Systems
Ekstrand, M. D., Das, A., Burke, R., & Díaz, F. (2022) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000079 · Cited by 198 · see citing works
Statistical Significance, p -Values, and the Reporting of Uncertainty
Imbens, G. W. (2021) · The Journal of Economic Perspectives · doi:10.1257/jep.35.3.157 · Cited by 135 · see citing works
Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement
Gordon, B. R., Moakler, R., & Zettelmeyer, F. (2022) · Marketing Science · doi:10.1287/mksc.2022.1413 · Cited by 108 · see citing works
Crafting and Assessing Design Science Research for Entrepreneurship
Dimov, D. P., MAULA, M. V. J., & Romme, A. G. L. (2022) · Entrepreneurship Theory and Practice · doi:10.1177/10422587221128271 · Cited by 98 · see citing works
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments
Muralidharan, K., Romero, M., & Wüthrich, K. (2020) · The Review of Economics and Statistics · doi:10.1162/rest_a_01317 · Cited by 96 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Rafieian, O., & Yoganarasimhan, H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1548-643520230000020004 · Cited by 77 · see citing works
Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments
Muralidharan, K., Romero, M., & Wüthrich, K. (2019) · National Bureau of Economic Research · doi:10.3386/w26562 · Cited by 75 · see citing works
Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology
Larsen, N., Stallrich, J. W., Sengupta, S., Deng, A., Kohavi, R., & Stevens, N. T. (2023) · The American Statistician · doi:10.1080/00031305.2023.2257237 · Cited by 73 · see citing works
Self-citationEvaluating conversational recommender systems
Jannach, D. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10229-x · Cited by 68 · see citing works
Quaquebeke, N. V., Salem, M., van Dijke, M., & Wenzel, R. (2022) · Organizational Psychology Review · doi:10.1177/20413866221097571 · Cited by 63 · see citing works
Design and Analysis of Switchback Experiments
Bojinov, I., Simchi‐Levi, D., & Zhao, J. (2022) · Management Science · doi:10.1287/mnsc.2022.4583 · Cited by 61 · see citing works
Jacob, C., Bourke, S., & Heuss, S. (2022) · JMIR Human Factors · doi:10.2196/41481 · Cited by 55 · see citing works
An AI decision‐making framework for business value maximization
Gudigantala, N., Madhavaram, S., & Bicen, P. (2023) · AI Magazine · doi:10.1002/aaai.12076 · Cited by 47 · see citing works
Designing a deep-tech venture builder to address grand challenges and overcome the valley of death
Romme, A. G. L., Bell, J., & Frericks, G. (2023) · Journal of Organization Design · doi:10.1007/s41469-023-00144-y · Cited by 44 · see citing works
Online Controlled Experiments and A/B Tests
Kohavi, R., & Longbotham, R. (2023) · Springer eBooks · doi:10.1007/978-1-4899-7502-7_891-2 · Cited by 39 · see citing works
Self-citationFrontiers: Generative AI and Personalized Video Advertisements
Kapoor, A., & Kumar, M. (2025) · Marketing Science · doi:10.1287/mksc.2023.0494 · Cited by 37 · see citing works
Online randomized controlled experiments at scale: lessons and extensions to medicine
Kohavi, R., Tang, D., XU, Y. N., Hemkens, L. G., & Ioannidis, J. P. A. (2020) · Trials · doi:10.1186/s13063-020-4084-y · Cited by 37 · see citing works
Self-citationFalse Discovery in A/B Testing
Berman, R., & den Bulte, C. V. (2020) · Management Science · doi:10.1287/mnsc.2021.4207 · Cited by 34 · see citing works
Pietsch, W. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108588676 · Cited by 32 · see citing works
Frontiers: Which Firms Gain from Digital Advertising? Evidence from a Field Experiment
Dai, W., Kim, H., & Luca, M. (2023) · Marketing Science · doi:10.1287/mksc.2023.1436 · Cited by 31 · see citing works
Leveraging artificial intelligence for enhanced cybersecurity: insights and innovations
Thapaliya, S., & Bokani, A. (2024) · SADGAMAYA. · doi:10.3126/sadgamaya.v1i1.66888 · Cited by 28 · see citing works
Jeunen, O., Potapov, I., & Ustimenko, A. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671687 · Cited by 28 · see citing works
Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms
Li, H., Zhao, G., Ramesh Johari, B., & Weintraub, G. Y. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512063 · Cited by 27 · see citing works
How to overcome the cost of a criminal record for getting hired
Santos, M. R., Jaynes, C. M., & Thomas, D. M. (2023) · Criminology · doi:10.1111/1745-9125.12345 · Cited by 27 · see citing works
Artificial Intelligence, Lean Startup Method, and Product Innovations
Wang, X., & Wu, L. (2025) · Management Science · doi:10.1287/mnsc.2022.03905 · Cited by 26 · see citing works
Kohavi, R., Deng, A., & Vermeer, L. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539160 · Cited by 26 · see citing works
Self-citationEinarsson, Á. M., Helles, R., & Lomborg, S. (2024) · Information Communication & Society · doi:10.1080/1369118x.2024.2334411 · Cited by 24 · see citing works
Augmenting adjusted plus-minus in soccer with FIFA ratings
Matano, F., Richardson, L. F., Pospisil, T., Politsch, C. A., & Qin, J. (2023) · Journal of Quantitative Analysis in Sports · doi:10.1515/jqas-2021-0005 · Cited by 24 · see citing works
Cocreating the ICU-PAUSE Tool for Intensive Care Unit–Ward Transitions
Santhosh, L., Rojas, J. C., Garcia, B., Thomashow, M., & Lyons, P. G. (2022) · ATS Scholar · doi:10.34197/ats-scholar.2021-0135in · Cited by 21 · see citing works
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
Fabijan, A., Dmitriev, P., Olsson, H. H., Bosch, J., Vermeer, L., & Lewis, D. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00009 · Cited by 21 · see citing works
The Ethics of Online Controlled Experiments (A/B Testing)
Polonioli, A., Ghioni, R., Greco, C., Juneja, P., Tagliabue, J., Watson, D., & Floridi, L. (2023) · Minds and Machines · doi:10.1007/s11023-023-09644-y · Cited by 20 · see citing works
Tight Concentrations and Confidence Sequences From the Regret of Universal Portfolio
Orabona, F., & Jun, K.-S. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3330187 · Cited by 19 · see citing works
Novelty and Primacy: A Long-Term Estimator for Online Experiments
Sadeghi, S., Gupta, S., Gramatovici, S., Lu, J., Ai, H., & Zhang, R. (2022) · Technometrics · doi:10.1080/00401706.2022.2124309 · Cited by 18 · see citing works
Recommending With, Not For: Co-Designing Recommender Systems for Social Good
Ekstrand, M. D., Razi, A., Sarcevic, A., Pera, M. S., Burke, R., & Wright, K. L. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3759261 · Cited by 17 · see citing works
O’Keefe, D. J. (2023) · Communication Methods and Measures · doi:10.1080/19312458.2023.2214949 · Cited by 17 · see citing works
The Importance of Being Causal
Bojinov, I., Chen, A., & Liu, M. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.3b87b6b0 · Cited by 17 · see citing works
Heidhues, P., Köster, M., & Kőszegi, B. (2023) · The Economic Journal · doi:10.1093/ej/ueac093 · Cited by 16 · see citing works
Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021
Balog, K., Maxwell, D., Thomas, P., & Zhang, S. (2021) · ACM SIGIR Forum · doi:10.1145/3527546.3527559 · Cited by 15 · see citing works
Yi, C., Jiang, Z. J., & Zhou, M. (2023) · Production and Operations Management · doi:10.1111/poms.14003 · Cited by 15 · see citing works
A Bias Correction Approach for Interference in Ranking Experiments
Goli, A., Lambrecht, A., & Yoganarasimhan, H. (2023) · Marketing Science · doi:10.1287/mksc.2022.0046 · Cited by 15 · see citing works
Toward Optimal Variance Reduction in Online Controlled Experiments
Jin, Y., & Ba, S. (2022) · Technometrics · doi:10.1080/00401706.2022.2142670 · Cited by 14 · see citing works
Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization
Candogan, O., Chen, C., & Niazadeh, R. (2023) · Management Science · doi:10.1287/mnsc.2021.02741 · Cited by 14 · see citing works
Design and Analysis of Switchback ExperimentsPreprint
Bojinov, I., Simchi‐Levi, D., & Zhao, J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3684168 · Cited by 14 · see citing works
Triggering the Personalization Backfire Effect: The Moderating Role of Situational Privacy Concern
Kim, H., & Han, S. (2025) · Behavioral Sciences · doi:10.3390/bs15101323 · Cited by 13 · see citing works
Multi-armed bandits in the wild: Pitfalls and strategies in online experiments
Mattos, D. I., Bosch, J., & Olsson, H. H. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.05.004 · Cited by 13 · see citing works
Nichifor, E., Lixandroiu, R. C., Chițu, I. B., Brătucu, G., Sumedrea, S., Maican, C. I., & Tecău, A. S. (2021) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer16060128 · Cited by 13 · see citing works
Kim, G. Y. K., Rostosky, R., Bishop, F. K., Watson, K., Prahalad, P., Vaidya, A., Lee, S. C. L., Diana, A., Beacock, C., Chu, B. K., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01319-x · Cited by 12 · see citing works
Use model averaging instead of model selection in pulsar timing
van Haasteren, R. (2024) · Monthly Notices of the Royal Astronomical Society Letters · doi:10.1093/mnrasl/slae108 · Cited by 12 · see citing works
Ye, Z., Zhang, Z., Zhang, D., Zhang, H., & Zhang, R. (2025) · Management Science · doi:10.1287/mnsc.2024.04625 · Cited by 12 · see citing works
Is designed data collection still relevant in the big data era?
Anderson‐Cook, C. M., & Lu, L. (2023) · Quality and Reliability Engineering International · doi:10.1002/qre.3326 · Cited by 11 · see citing works
Stevens, N. T., & Hagar, L. (2021) · The American Statistician · doi:10.1080/00031305.2021.2000495 · Cited by 11 · see citing works
Semi-parametric estimation of treatment effects in randomised experiments
Athey, S. C., Bickel, P. J., Chen, A., Imbens, G. W., & Pollmann, M. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad072 · Cited by 11 · see citing works
Long-Term Effect Estimation with Surrogate Representation
Cheng, L., Guo, R., & Liu, H. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441719 · Cited by 11 · see citing works
Unbiased experiments in congested networks
Spang, B., Hannan, V., Kunamalla, S., Huang, T.-Y., McKeown, N., & Ramesh Johari, B. (2021) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3487552.3487851 · Cited by 11 · see citing works
Feature Selection Methods for Uplift Modeling and Heterogeneous Treatment Effect
Zhao, Z., Zhang, Y., Harinen, T., & Yung, M. (2022) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-08337-2_19 · Cited by 11 · see citing works
Which Firms Gain from Digital Advertising? Evidence from a Field ExperimentPreprint
Dai, D., Kim, H., & Luca, M. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2847084 · Cited by 11 · see citing works
Profiles in the Teaching of Experimental Design and Analysis
Smucker, B. J., Stevens, N. T., Asscher, J., & Goos, P. (2023) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2023.2205907 · Cited by 10 · see citing works
Engineering for a science-centric experimentation platform
Diamantopoulos, N., Wong, J., Mattos, D. I., Gerostathopoulos, I., Wardrop, M. P., Mao, T., & McFarland, C. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3377813.3381349 · Cited by 10 · see citing works
Testing software’s changing features with environment-driven abstraction identification
Peng, Z., Rathod, P., Niu, N., Bhowmik, T., Liu, H., Shi, L., & Jin, Z. G. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00390-8 · Cited by 10 · see citing works
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 9 · see citing works
A Common Misassumption in Online Experiments with Machine Learning Models
Jeunen, O. (2023) · ACM SIGIR Forum · doi:10.1145/3636341.3636358 · Cited by 9 · see citing works
Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective
Zhao, J., & Zhou, Z. (2022) · Management Science · doi:10.1287/mnsc.2023.02184 · Cited by 9 · see citing works
Causal message-passing for experiments with unknown and general network interference
Faradonbeh, M. S. S., & Bayati, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322232121 · Cited by 9 · see citing works
Zou, T., Xiong, F., Li, S., & Zhang, W. (2025) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3415500 · Cited by 9 · see citing works
Supply Chain Transparency and Sustainability
Kraft, T., Yu, J. V., & Zheng, Y. (2023) · Foundations and Trends® in Technology Information and Operations Management · doi:10.1561/0200000107 · Cited by 8 · see citing works
Kwon, H., Kim, J., Lim, H.-H., Kim, J. M., Kim, Y., Sunwoo, C., & Hwang, D. (2025) · IEEE Access · doi:10.1109/access.2025.3588480 · Cited by 8 · see citing works
A learning health system approach to COVID ‐19 exposure notification system rollout
Tai‐Seale, M., May, N., Sitapati, A. M., & Longhurst, C. (2021) · Learning Health Systems · doi:10.1002/lrh2.10290 · Cited by 8 · see citing works
Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments
Ham, D. W., Lindon, M., Tingley, M. P., & Bojinov, I. (2026) · Management Science · doi:10.1287/mnsc.2024.08235 · Cited by 8 · see citing works
It’s time to close the experimentation gap in advertising: Confronting myths surrounding ad testing
Campbell, C., Runge, J., Bates, K., Haefele, S., & Jayaraman, N. (2021) · Business Horizons · doi:10.1016/j.bushor.2021.05.004 · Cited by 8 · see citing works
Variance Reduction in Ratio Metrics for Efficient Online Experiments
Baweja, S., Pokharna, N., Ustimenko, A., & Jeunen, O. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_34 · Cited by 8 · see citing works
Quantifying the Value of Iterative ExperimentationPreprint
Mao, J., & Bojinov, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02334 · Cited by 8 · see citing works
Baas, S., Jacko, P., & Villar, S. S. (2025) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2025.108207 · Cited by 7 · see citing works
Locally Optimal Design for A/B Tests in the Presence of Covariates and Network Dependence
Zhang, Q., & Kang, L. (2022) · Technometrics · doi:10.1080/00401706.2022.2046169 · Cited by 7 · see citing works
Amsalem, D., Martin, A., Dixon, L. B., & Haim‐Nachum, S. (2025) · JAACAP Open · doi:10.1016/j.jaacop.2024.12.004 · Cited by 7 · see citing works
Shabalina, O., Nelson, M. R., & Wise, K. R. (2025) · Journal of Current Issues & Research in Advertising · doi:10.1080/10641734.2024.2441958 · Cited by 7 · see citing works
Detecting Interference in Online Controlled Experiments with Increasing Allocation
Han, K. W., Li, S., Mao, J., & Wu, H. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599308 · Cited by 7 · see citing works
Online Advertising Incrementality Testing And Experimentation
Barajas, J., Bhamidipati, N. L., & Shanahan, J. G. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3470819 · Cited by 6 · see citing works
Containerized Microservices for Mobile Applications Deployed on Cloud Systems
Jordanov, J., Simeonidis, D., & Petrov, P. (2024) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v18i10.45929 · Cited by 6 · see citing works
Dealing with Ratio Metrics in A/B Testing at the Presence of Intra-user Correlation and Segments
Nie, K., Kong, Y., Yuan, T. T., & Burke, P. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62008-0_39 · Cited by 6 · see citing works
Pricing Experimental Design: Causal Effect, Expected Revenue and Tail Risk
Simchi‐Levi, D., & Wang, C. (2025) · Management Science · doi:10.1287/mnsc.2023.03209 · Cited by 5 · see citing works
UX debt in an agile development process: evidence and characterization
Rodríguez, A., Gardey, J. C., Grigera, J., Rossi, G. H., & Garrido, A. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09652-2 · Cited by 5 · see citing works
Continuous Monitoring of Data in Online Randomized Experiments
Bondarenko, Y. (2023) · International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT) · doi:10.1109/3ict60104.2023.10391343 · Cited by 5 · see citing works
Kohavi, R., & Chen, N. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671631 · Cited by 5 · see citing works
Self-citationLocal Search Quality: Comment on “Product Quality and Entering Through Tying: Experimental Evidence”
Millet, F. C., Lewis, S., & Stoddart, P. (2022) · Management Science · doi:10.1287/mnsc.2021.4276 · Cited by 5 · see citing works
Automated Online Experiment-Driven Adaptation–Mechanics and Cost Aspects
Gerostathopoulos, I., Plášil, F., Prehofer, C., Thomas, J., & Bischl, B. (2021) · IEEE Access · doi:10.1109/access.2021.3071809 · Cited by 5 · see citing works
Efficient Switchback Experiments with Surrogate Variables: Estimation and Experimental Design
Chen, H., & Simchi‐Levi, D. (2025) · Management Science · doi:10.1287/mnsc.2023.03818 · Cited by 5 · see citing works
Editors’ Parting Thoughts: Retrospective and Prospective
Rangaswamy, A., & Gensler, S. (2024) · Journal of Interactive Marketing · doi:10.1177/10949968241258574 · Cited by 5 · see citing works
Reliably estimating the impact of an active control strategy in a building
Raftery, P., Zou, A., Parkinson, T., & Hancock, G. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.111134 · Cited by 4 · see citing works
Experimentation in Organizations: An Integrative Review
Corbo, L., Katila, R., & Vlačić, B. (2026) · Academy of Management Annals · doi:10.5465/annals.2024.0287 · Cited by 4 · see citing works
Nonstationary A/B Tests: Optimal Variance Reduction, Bias Correction, and Valid Inference
Wu, Y., Zheng, Z., Zhang, G., Zhang, Z., & Wang, C. (2024) · Management Science · doi:10.1287/mnsc.2022.01205 · Cited by 4 · see citing works
Trippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 4 · see citing works
Towards Continuous Experiment-Driven MLOps
Rajenthiram, K., Abdullah, M., Gerostathopoulos, I., Hnětynka, P., Bureš, T., Pons, G., Bilalli, B., & Queralt, A. (2025) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00018 · Cited by 4 · see citing works
Understanding and Evaluating Search Experience
Stone, M. V. (2022) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s01166ed1v01y202202icr077 · Cited by 4 · see citing works
Policy learning “without” overlap: Pessimism and generalized empirical Bernstein’s inequality
Jin, Y., Ren, Z., Yang, Z., & Wang, Z. (2025) · The Annals of Statistics · doi:10.1214/25-aos2511 · Cited by 4 · see citing works
A Real-World Implementation of Unbiased Lift-based Bidding System
Moriwaki, D., Hayakawa, Y., Matsui, A., Saito, Y., Munemasa, I., & Shibata, M. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671800 · Cited by 4 · see citing works
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 3 · see citing works
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 3 · see citing works
- Authors
- Kohavi, R., Tang, D., & Xu, Y.
- Affiliations
- Microsoft (United States) (US)
- Google (United States) (US)
- LinkedIn (United States) (US)
- ORCID
- Kohavi, Ron
- Venue
- Cambridge University Press eBooks
- Publisher
- Cambridge University Press
- Published
- 2020-03-13
- OpenAlex
- W3011423189
- Cited by
- 297
- References
- 235
- Citations by year
- 2026: 64 · 2025: 53 · 2024: 46 · 2023: 39 · 2022: 34 · 2021: 19 · 2020: 38 · 2019: 3 · 2016: 1
- FWCI
- 16.61
- Type
- book
- Language
- en
- Open access
- closed
- Locations
- Cambridge University Press eBooks (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Optimal Experimental Design Methods
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- A/B testing, experimentation, data-driven decision making, hypothesis testing, statistical significance, residual effects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
