150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
Abstract
Booking.com is the world's largest online travel agent where millions of guests find their accommodation and millions of accommodation providers list their properties including hotels, apartments, bed and breakfasts, guest houses, and more. During the last years we have applied Machine Learning to improve the experience of our customers and our business. While most of the Machine Learning literature focuses on the algorithmic or mathematical aspects of the field, not much has been published about how Machine Learning can deliver meaningful impact in an industrial environment where commercial gains are paramount. We conducted an analysis on about 150 successful customer facing applications of Machine Learning, developed by dozens of teams in Booking.com, exposed to hundreds of millions of users worldwide and validated through rigorous Randomized Controlled Trials. Following the phases of a Machine Learning project we describe our approach, the many challenges we found, and the lessons we learned while scaling up such a complex technology across our organization. Our main conclusion is that an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by Machine Learning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Currently, the options for meeting performance SLOs during high-demand scenarios are either to accept degraded throughput or latency, or to scale up hardware resources and incur increased costs.
“The authors noted that currently, the options are to either accept degraded throughput or latency during high-demand scenarios, or to scale up the hardware resources (as suggested by [6]) and thereby incurring increased costs.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
Monitoring ML models is challenging but is required in order to detect performance degradations and build confidence in the models.
“Monitoring ML models is challenging [6,28,41], but is required in order to detect performance degradations and build confidence in the models [7,10].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
Serving-related challenges are some of the ones most often reported in the literature, usually from a performance perspective.
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
There is a statistically significant negative correlation between latency and conversion rate at Booking.com.
“[6] found that there is a statistically significant negative correlation between latency and conversion rate at Booking.com, highlighting the business impact of serving performance.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models.
“Achieving a low inference latency and high throughput is widely reported as a challenge in serving ML models [6,7,12,28,34].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
One solution used in practice is to look at the distribution of predictions and determine if there is a significant deviation from what the response distribution of an ideal model would look like.
“One solution used in practice is to look at the distribution of predictions and determine if there is a significant deviation from what the response distribution of an ideal model would look like [6].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
After an inference is made, the true label may not be available for an extended period of time, making it difficult to monitor prediction quality.
“After an inference is made, the true label may not be available for an extended period of time, making it difficult to monitor prediction quality [6].”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [6]
Cites
Estimation of Regression Coefficients When Some Regressors Are Not Always Observed
Robins, J. M., Rotnitzky, A. G., & Zhao, L. P. (1994) · Journal of the American Statistical Association · doi:10.1080/01621459.1994.10476818 · Cited by 2,803 · see citing works
Mitzias, P., Riga, M., Waddington, S. N., Kontopoulos, E., Μεδίτσκος, Γ., Laurenson, P., & Kompatsiaris, I. Y. (2015) · Cited by 1,240 · see citing works
Doubly Robust Policy Evaluation and LearningPreprint
Dudı́k, M., Langford, J. C., & Li, L. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.4601 · Cited by 396 · see citing works
Recommender systems — beyond matrix completion
Jannach, D., Resnick, P., Tuzhilin, A., & Zanker, M. (2016) · Communications of the ACM · doi:10.1145/2891406 · Cited by 180 · see citing works
Impact of response latency on user behavior in web search
Arapakis, I., Bai, X., & Cambazoglu, B. B. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609627 · Cited by 151 · see citing works
Learning from Positive and Unlabeled Examples with Different Data Distributions
Li, X., & Liu, B. (2005) · Lecture notes in computer science · doi:10.1007/11564096_24 · Cited by 115 · see citing works
Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments
Deng, A., & Hu, V. (2015) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2684822.2685307 · Cited by 57 · see citing works
Bayes Error Rate Estimation Using Classifier Ensembles
Tumer, K., & Ghosh, J. (2003) · International Journal of Smart Engineering System Design · doi:10.1080/10255810305042 · Cited by 27 · see citing works
Democratizing online controlled experiments at Booking.comPreprint
Kaufman, R. L., Pitchforth, J., & Vermeer, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.08217 · Cited by 21 · see citing works
Toward a full-scale neural machine translation in production: the Booking.com use casePreprint
Levin, P., Dhanuka, N., Khalil, T., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.05820 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Challenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638 · see citing works
Engineering AI Systems: A Research Agenda
Bosch, J., Olsson, H. H., & Crnkovic, I. (2021) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-7998-5101-1.ch001 · Cited by 108 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
Data Science Project Barriers—A Systematic Review
Labarrère, N., Costa, L., & Lima, R. M. (2025) · Data · doi:10.3390/data10080132 · Cited by 2 · see citing works
- Authors
- Bernardi, L., Mavridis, T., & Estevez, P.
- Affiliations
- Booking.com, Amsterdam, Netherlands
- Venue
- Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2019-07-25
- Pages
- 1743-1751
- OpenAlex
- W2951641587
- Cited by
- 95
- References
- 10
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 5 · 2023: 9 · 2022: 16 · 2021: 22 · 2020: 19 · 2019: 3 · 2018: 1 · 2017: 1 · 2015: 2 · 2013: 1 · 2012: 1
- FWCI
- 5.74
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- online travel agencies, Booking.com, iterative development, randomized controlled trials, lessons learned
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
