Targeting Prospective Customers: Robustness of Machine-Learning Methods to Typical Data Challenges

Simester, Duncan; Timoshenko, Artem; Zoumpoulis, Spyros I.

doi:10.1287/mnsc.2019.3308

Cited by 133Customer churn and segmentationConsumer Market Behavior and PricingForecasting Techniques and Applications

Targeting Prospective Customers: Robustness of Machine-Learning Methods to Typical Data Challenges

Open access

Simester, Duncan, Timoshenko, Artem, Zoumpoulis, Spyros I.·2020·Management Science

DOI 10.1287/mnsc.2019.3308

Open this paper in CoChat — free account

Abstract

We investigate how firms can use the results of field experiments to optimize the targeting of promotions when prospecting for new customers. We evaluate seven widely used machine-learning methods using a series of two large-scale field experiments. The first field experiment generates a common pool of training data for each of the seven methods. We then validate the seven optimized policies provided by each method together with uniform benchmark policies in a second field experiment. The findings not only compare the performance of the targeting methods, but also demonstrate how well the methods address common data challenges. Our results reveal that when the training data are ideal, model-driven methods perform better than distance-driven methods and classification methods. However, the performance advantage vanishes in the presence of challenges that affect the quality of the training data, including the extent to which the training data captures details of the implementation setting. The challenges we study are covariate shift, concept shift, information loss through aggregation, and imbalanced data. Intuitively, the model-driven methods make better use of the information available in the training data, but the performance of these methods is more sensitive to deterioration in the quality of this information. The classification methods we tested performed relatively poorly. We explain the poor performance of the classification methods in our setting and describe how the performance of these methods could be improved. This paper was accepted by Matthew Shum, marketing.

Integrity signals

Registry

Crossref · 10.1287/mnsc.2019.3308

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Various thinking AI can be used for targeting.
“Existing studies show that various thinking AI can be used for this purpose.”
From A strategic framework for artificial intelligence in marketing (2021) · cites this paper as Simester et al. 2020

Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 16,037 · see citing works
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · see citing works
Approximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188 · see citing works
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · see citing works
Double/debiased machine learning for treatment and structural parameters
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. K., & Robins, J. M. (2017) · Econometrics Journal · doi:10.1111/ectj.12097 · Cited by 3,196 · see citing works
Feature selection: evaluation, application, and small sample performance
Jain, A. K., & Zongker, D. E. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.574797 · Cited by 2,186 · see citing works
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · see citing works
Classification in the Presence of Label Noise: A Survey
Frenay, B., & Verleysen, M. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2292894 · Cited by 1,777 · see citing works
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678 · see citing works
Integrating structured biological data by Kernel Maximum Mean Discrepancy
Borgwardt, K. M., Gretton, A., Rasch, M. J., Kriegel, H.-P., Schölkopf, B., & Smola, A. J. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl242 · Cited by 1,674 · see citing works
Decision-Making Under Uncertainty: Capturing Dynamic Brand Choice Processes in Turbulent Consumer Goods Markets
Erdem, T., & Keane, M. P. (1996) · Marketing Science · doi:10.1287/mksc.15.1.1 · Cited by 1,230 · see citing works
Predicting breast cancer survivability: a comparison of three data mining methods
Delen, D., Walker, G., & Kadam, A. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.07.002 · Cited by 1,215 · see citing works
A unifying view on dataset shift in classification
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012) · Pattern Recognition · doi:10.1016/j.patcog.2011.06.019 · Cited by 1,029 · see citing works
Imbalanced Learning: Foundations, Algorithms, and Applications
He, H.-B., & Ma, Y. (2013) · Wiley eBooks · doi:10.1002/9781118646106 · Cited by 917 · see citing works
Learning and evaluating classifiers under sample selection bias
Zadrozny, B. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015425 · Cited by 800 · see citing works
Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · see citing works
Multivariate Generalizations of the Wald-Wolfowitz and Smirnov Two-Sample Tests
Friedman, J. H., & Rafsky, L. C. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344722 · Cited by 695 · see citing works
Incremental Learning from Noisy Data
Schlimmer, J. C., & Granger, R. (1986) · Machine Learning · doi:10.1023/a:1022810614389 · Cited by 652 · see citing works
Learning a Mahalanobis distance metric for data clustering and classification
Xiang, S., Nie, F., & Zhang, C. (2008) · Pattern Recognition · doi:10.1016/j.patcog.2008.05.018 · Cited by 629 · see citing works
Equivalence of distance-based and RKHS-based statistics in hypothesis testing
Sejdinovic, D., Sriperumbudur, B., Gretton, A., & Fukumizu, K. (2013) · The Annals of Statistics · doi:10.1214/13-aos1140 · Cited by 582 · cited 1× · see citing works
“As we will discuss, this test is distribution-free, computationally ef ﬁ cient, and performs strongly across a variety of applications (Gretton et al. 2012, Sejdinovic et al. 2013, Li et al. 2015).”
Method
Tracking the Best Expert
Herbster, M., & Warmuth, M. K. (1998) · Machine Learning · doi:10.1023/a:1007424614876 · Cited by 559 · see citing works
Bioinformatics: The Machine Learning Approach
Leunissen, J. A. M. (2002) · Briefings in Bioinformatics · doi:10.1093/bib/3.3.321 · Cited by 477 · see citing works
Is Aggregation Necessarily Bad?
Grunfeld, Y., & Griliches, Z. (1960) · The Review of Economics and Statistics · doi:10.2307/1926089 · Cited by 432 · see citing works
Double/Debiased/Neyman Machine Learning of Treatment Effects
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., & Newey, W. K. (2017) · American Economic Review · doi:10.1257/aer.p20171038 · Cited by 371 · see citing works
Segmentation approaches in data-mining: A comparison of RFM, CHAID, and logistic regression
McCarty, J. A., & Hastak, M. (2007) · Journal of Business Research · doi:10.1016/j.jbusres.2006.06.015 · Cited by 306 · see citing works
Planning Media Schedules in the Presence of Dynamic Advertising Quality
Naik, P., Mantrala, M., & Sawyer, A. G. (1998) · Marketing Science · doi:10.1287/mksc.17.3.214 · Cited by 302 · see citing works
Machine Learning in Non-Stationary Environments
Sugiyama, M., & Kawanabe, M. (2012) · The MIT Press eBooks · doi:10.7551/mitpress/9780262017091.001.0001 · Cited by 222 · see citing works
Extracting Hidden Context
Harries, M. B., Sammut, C., & Horn, K. (1998) · Machine Learning · doi:10.1023/a:1007420529897 · Cited by 179 · see citing works
Tracking Drifting Concepts By Minimizing Disagreements
Helmbold, D. P., & Long, P. M. (1994) · Machine Learning · doi:10.1023/a:1022694620923 · Cited by 161 · see citing works
Econometric Analysis of Aggregation in the Context of Linear Prediction Models
Pesaran, M. H., Pierse, R. G., & Kumar, M. (1989) · Econometrica · doi:10.2307/1913775 · Cited by 159 · see citing works
Efficient optimization of support vector machine learning parameters for unbalanced datasets
Eitrich, T., & Lang, B. (2005) · Journal of Computational and Applied Mathematics · doi:10.1016/j.cam.2005.09.009 · Cited by 143 · see citing works
Permutation tests for equality of distributions in high-dimensional settings
Hall, P. M. (2002) · Biometrika · doi:10.1093/biomet/89.2.359 · Cited by 135 · see citing works
On the dataset shift problem in software engineering prediction models
Turhan, B. (2011) · Empirical Software Engineering · doi:10.1007/s10664-011-9182-8 · Cited by 122 · see citing works
Efficient resampling methods for training support vector machines with imbalanced datasets
Batuwita, R., & Palade, V. (2010) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596787 · Cited by 116 · see citing works
Do Household Scanner Data Provide Representative Inferences from Brand Choices: A Comparison with Store Data
Gupta, S., Chintagunta, P. K., Kaul, A., & Wittink, D. R. (1996) · Journal of Marketing Research · doi:10.1177/002224379603300401 · Cited by 107 · see citing works
Tracking the Best Disjunction
Auer, P. L., & Warmuth, M. K. (1998) · Machine Learning · doi:10.1023/a:1007472513967 · Cited by 106 · see citing works
Estimation and Prediction from Aggregate Data when Aggregates are Measured More Accurately than Their Components
Aigner, D. J., & Goldfeld, S. M. (1974) · Econometrica · doi:10.2307/1913689 · Cited by 102 · see citing works
The Normative Impact of Consumer Price Expectations for Multiple Brands on Consumer Purchase Behavior
Krishna, A. (1992) · Marketing Science · doi:10.1287/mksc.11.3.266 · Cited by 93 · see citing works
The Impact of Dealing Patterns on Purchase Behavior
Krishna, A. (1994) · Marketing Science · doi:10.1287/mksc.13.4.351 · Cited by 92 · see citing works
Real-Data Comparison of Data Mining Methods in Prediction of Diabetes in Iran
Tapak, L., Mahjub, H., Hamidi, O., & Poorolajal, J. (2013) · Healthcare Informatics Research · doi:10.4258/hir.2013.19.3.177 · Cited by 83 · see citing works
Advances in Knowledge Discovery and Data Mining
(2003) · Lecture notes in computer science · doi:10.1007/3-540-36175-8 · Cited by 76 · see citing works
“The Best Price You'll Ever Get”: The 2005 Employee Discount Pricing Promotions in the U.S. Automobile Industry
Busse, M. R., Simester, D. I., & Zettelmeyer, F. (2009) · Marketing Science · doi:10.1287/mksc.1090.0516 · Cited by 75 · see citing works
Self-citation
Comparison of predictive modeling approaches for 30-day all-cause non-elective readmission risk
Tong, L.-P., Erdmann, C., Daldalian, M., Li, J., & Esposito, T. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0128-0 · Cited by 61 · see citing works
Should Aggregation Prior to Estimation be the Rule?
Edwards, J. B., & Orcutt, G. H. (1969) · The Review of Economics and Statistics · doi:10.2307/1926432 · Cited by 60 · see citing works
Handling Local Concept Drift with Dynamic Integration of Classifiers: Domain of Antibiotic Resistance in Nosocomial Infections
Tsymbal, A., Pechenizkiy, M., Cunningham, P., & Puuronen, S. J. (2006) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2006.94 · Cited by 54 · see citing works
Assessing the Impact of Changing Environments on Classifier Performance
Aláiz-Rodríguez, R., & Japkowicz, N. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-68825-9_2 · Cited by 51 · see citing works
A comparison and an exploration of the forecasting accuracy of a loglinear model at different levels of aggregation
Foekens, E. W., Leeflang, P. S. H., & Wittink, D. R. (1994) · International Journal of Forecasting · doi:10.1016/0169-2070(94)90005-1 · Cited by 49 · see citing works
On the Asymptotic Properties of a Nonparametric $L_1$ -Test Statistic of Homogeneity
Gérard Biau, A., & Györfi, L. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.856979 · Cited by 46 · see citing works
Comparison of customer response models
Olson, D. L., Cao, Q., Gu, C., & Lee, D. (2009) · Service Business · doi:10.1007/s11628-009-0064-8 · Cited by 37 · see citing works
Online mass flow prediction in CFB boilers with explicit detection of sudden concept drift
Pechenizkiy, M., Bakker, J., Žliobaitė, I., Ivannikov, A., & Kärkkäinen, T. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809423 · Cited by 36 · see citing works
Drifting Concepts as Hidden Factors in Clinical Studies
Kukar, M. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39907-0_49 · Cited by 35 · see citing works
A support vector machine (SVM) approach to imbalanced datasets of customer responses: comparison with other customer response models
Kim, G., Chae, B. K., & Olson, D. L. (2012) · Service Business · doi:10.1007/s11628-012-0147-9 · Cited by 34 · see citing works
Asymptotic Bayesian generalization error when training and test distributions are different
Yamazaki, K., Kawanabe, M., Watanabe, S., Sugiyama, M., & Müller, K. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273632 · Cited by 33 · see citing works
Statistical Mechanics of Online Learning of Drifting Concepts: A Variational Approach
Vicente, R., Kinouchi, O., & Caticha, N. (1998) · Machine Learning · doi:10.1023/a:1007428731714 · Cited by 30 · see citing works
Robust Sensor Fusion: Analysis and Application to Audio Visual Speech Recognition
Movellan, J. R., & Mineiro, P. (1998) · Machine Learning · doi:10.1023/a:1007468413059 · Cited by 29 · see citing works
Aggregation Bias in Sponsored Search Data: The Curse and the Cure
Abhishek, V., Hosanagar, K., & Fader, P. S. (2015) · Marketing Science · doi:10.1287/mksc.2014.0884 · Cited by 29 · see citing works
Adaptive Bayes for a Student Modeling Prediction Task Based on Learning Styles
Castillo, G., da Gama, J. M. P., & Breda, A. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44963-9_44 · Cited by 28 · see citing works
Estimating the Effects of Consumer Incentive Programs on Domestic Automobile Sales
Thompson, P. A., & Noordewier, T. G. (1992) · Journal of Business and Economic Statistics · doi:10.1080/07350015.1992.10509916 · Cited by 28 · see citing works
A Comparison of Sales Response Predictions From Demand Models Applied to Store-Level versus Panel Data
Andrews, R. L., Currim, I. S., & Leeflang, P. S. H. (2010) · Journal of Business and Economic Statistics · doi:10.1198/jbes.2010.07225 · Cited by 22 · see citing works
Imputation methods to deal with missing values when data mining trauma injury data
Penny, K. I., & Chesney, T. (2006) · doi:10.1109/iti.2006.1708480 · Cited by 15 · see citing works
Tracking Drifting Concepts Using Random Examples
Helmbold, D. P., & Long, P. M. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-213-7.50006-7 · Cited by 14 · see citing works
Simulation and Aggregation: A Reconsideration
Aigner, D. J., & Goldfeld, S. M. (1973) · The Review of Economics and Statistics · doi:10.2307/1928002 · Cited by 13 · see citing works
Rejoinder: Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000079 · Cited by 10 · see citing works
Guest Editors'Introduction
Widmer, G., & Kubát, M. (1998) · Machine Learning · doi:10.1023/a:1007456328988 · Cited by 6 · see citing works
A comparison of data mining methods in microfinance
Wu, J. X., Vadera, S., Dayson, K. T., Burridge, D., & Clough, I. (2010) · IEEE International Conference on Information and Financial Engineering · doi:10.1109/icife.2010.5609408 · Cited by 4 · see citing works
Editorial
Dietterich, T. G. (1992) · Machine Learning · doi:10.1007/bf00992860 · Cited by 1 · see citing works

Flagged first, then by citations

Sorted by citations

A strategic framework for artificial intelligence in marketing
Huang, M.-H., & Rust, R. T. (2021) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-020-00749-9 · Cited by 2,059 · see citing works
“…machine learning and causal forests (Chen et al. 2020), optimizing promotion targeting for new customers using various machine learning methods (Simester et al. 2020), identifying the best targets for proactive churn programs from field experimental data using machine learning techniques…”
Method
Artificial intelligence in marketing: Systematic review and future research direction
Verma, S., Sharma, R., Deb, S., & Maitra, D. (2021) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2020.100002 · Cited by 905 · see citing works
FinTech as a Game Changer: Overview of Research Frontiers
Hendershott, T. J., Zhang, X., Zhao, J. L., & Zheng, Z. (2021) · Information Systems Research · doi:10.1287/isre.2021.0997 · Cited by 205 · see citing works
Role of artificial intelligence in marketing strategies and performance
Wu, C., & Monfort, A. (2022) · Psychology and Marketing · doi:10.1002/mar.21737 · Cited by 130 · see citing works
A review of spam email detection: analysis of spammer strategies and the dataset shift problem
Jáñez-Martino, F., Aláiz-Rodríguez, R., González‐Castro, V., Fidalgo, E., & Alegre, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10195-4 · Cited by 125 · see citing works
Machine learning in business and finance: a literature review and research opportunities
Gao, H., Kou, G., Liang, H., Zhang, H., Chao, X., Li, C., & Dong, Y. (2024) · Financial Innovation · doi:10.1186/s40854-024-00629-z · Cited by 120 · see citing works
Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation
Hitsch, G. J., Misra, S., & Zhang, W. W. (2018) · Quantitative Marketing and Economics · doi:10.1007/s11129-023-09278-5 · Cited by 93 · see citing works
Algorithmic fairness in business analytics: Directions for research and practice
De‐Arteaga, M., Feuerriegel, S., & Saar‐Tsechansky, M. (2022) · Production and Operations Management · doi:10.1111/poms.13839 · Cited by 90 · see citing works
Product Choice with Large Assortments: A Scalable Deep-Learning Model
Gabel, S., & Timoshenko, A. (2021) · Management Science · doi:10.1287/mnsc.2021.3969 · Cited by 77 · see citing works
Self-citation
Does the Sharing Economy Technology Disrupt Incumbents? Exploring the Influences of Mobile Digital Freight Matching Platforms on Road Freight Logistics Firms
Zhou, Z., & Wan, X. (2021) · Production and Operations Management · doi:10.1111/poms.13491 · Cited by 72 · see citing works
AI and Personalization
Rafieian, O., & Yoganarasimhan, H. (2023) · Emerald Group Publishing Limited eBooks · doi:10.1108/s1548-643520230000020004 · Cited by 77 · see citing works
Artificial intelligence and empirical consumer research: A topic modeling analysis
Vaid, S., Puntoni, S., & Khodr, A. (2023) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114110 · Cited by 71 · see citing works
Tourism demand forecasting and tourists’ search behavior: evidence from segmented Baidu search volume
Yang, Y., Guo, J., & Sun, S. (2021) · Data Science and Management · doi:10.1016/j.dsm.2021.10.002 · Cited by 57 · see citing works
Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments
Simester, D. I., Timoshenko, A., & Zoumpoulis, S. I. (2020) · Management Science · doi:10.1287/mnsc.2019.3379 · Cited by 54 · see citing works
Self-citation
Color effects on AI influencers’ product recommendations
Chan, K. W., Septianto, F., Kwon, J., & Kamal, R. S. (2023) · European Journal of Marketing · doi:10.1108/ejm-03-2022-0185 · Cited by 55 · see citing works
Optimal Price Targeting
Smith, A. N., Seiler, S., & Aggarwal, I. (2022) · Marketing Science · doi:10.1287/mksc.2022.1387 · Cited by 44 · see citing works
Targeting for Long-Term Outcomes
Yang, J., Eckles, D., Dhillon, P. S., & Aral, S. (2023) · Management Science · doi:10.1287/mnsc.2023.4881 · Cited by 44 · see citing works
Managing change when integrating artificial intelligence (AI) into the retail value chain: The AI implementation compass
Robertson, J., Botha, E., Oosthuizen, K., & Montecchi, M. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115198 · Cited by 57 · see citing works
Hyper-Personalization
Desai, D. (2021) · Advances in marketing, customer relationship management, and e-services book series · doi:10.4018/978-1-7998-7959-6.ch003 · Cited by 40 · see citing works
Informational vs. emotional B2B firm-generated-content on social media engagement: Computerized visual and textual content analysis
Gu, W., Chan, K. W., Kwon, J., Dhaoui, C., & Septianto, F. (2023) · Industrial Marketing Management · doi:10.1016/j.indmarman.2023.04.012 · Cited by 34 · see citing works
Substitution and complementarity between human and artificial intelligence: a dynamic capabilities view
Siaw, C. A., & Ali, W. (2024) · Journal of Managerial Psychology · doi:10.1108/jmp-06-2024-0398 · Cited by 26 · see citing works
Using machine learning to analyze longitudinal data: A tutorial guide and best‐practice recommendations for social science researchers
Sheetal, A., Jiang, Z., & Milia, L. V. D. (2022) · Applied Psychology · doi:10.1111/apps.12435 · Cited by 26 · see citing works
Will they take this offer? A machine learning price elasticity model for predicting upselling acceptance of premium airline seating
Thirumuruganathan, S., Emadi, N. A., Jung, S., Salminen, J. O., Robillos, D. R., & Jansen, B. J. J. (2023) · Information & Management · doi:10.1016/j.im.2023.103759 · Cited by 26 · see citing works
Testing machine learning systems in real estate
Wan, W. X., & Lindenthal, T. (2022) · Real Estate Economics · doi:10.1111/1540-6229.12416 · Cited by 24 · see citing works
Out-of-time cross-validation strategies for classification in the presence of dataset shift
Maldonado, S., López, J., & Iturriaga, A. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02735-2 · Cited by 24 · see citing works
Data augmentation for numerical data from manufacturing processes: an overview of techniques and assessment of when which techniques work
Ekwaro-Osire, H., Ponugupati, S. L., Noman, A. A., Bode, D., & Thoben, K. (2025) · Industrial Artificial Intelligence · doi:10.1007/s44244-024-00021-x · Cited by 22 · see citing works
Incentive Misalignments in Programmatic Advertising: Evidence from a Randomized Field Experiment
Frick, T. W., Belo, R., & Telang, R. (2022) · Management Science · doi:10.1287/mnsc.2022.4438 · Cited by 22 · see citing works
Machine Learning Methods for Data-Driven Demand Estimation and Assortment Planning Considering Cross-Selling and Substitutions
Chen, Z.-Y., Fan, Z., & Sun, M. (2022) · INFORMS journal on computing · doi:10.1287/ijoc.2022.1251 · Cited by 25 · see citing works
Artificial Intelligence in Marketing: Literature Review and Future Research Agenda
(2023) · Journal of System and Management Sciences · doi:10.33168/jsms.2024.0108 · Cited by 22 · see citing works
Understanding retail exclusion and promoting an inclusive customer experience at transforming service encounters
Lu, F., & Sinha, J. (2023) · Journal of Consumer Affairs · doi:10.1111/joca.12529 · Cited by 23 · see citing works
Targeted aspect-based emotion analysis to detect opportunities and precaution in financial Twitter messages
García-Méndez, S., de Arriba-Pérez, F., Barros-Vila, A., & González-Castaño, F. J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119611 · Cited by 19 · see citing works
AI and Its Implications for Organisations
Sharma, M., & Biros, D. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83909-812-320211001 · Cited by 12 · see citing works
Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical Approach
Wang, Y., Tao, L., & Zhang, X. X. (2024) · Marketing Science · doi:10.1287/mksc.2022.0238 · Cited by 24 · see citing works
"What Went Wrong? Identifying Risk Factors for Popular Negative Consequences in AI"
Sharma, M., Biros, D. P., Baham, C., & Biros, J. (2024) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00203 · Cited by 15 · see citing works
Identifying Influencing Factors for Data Transactions: A Case Study from Shanghai Data Exchange
Tang, Q., Shao, Z., Huang, L., Yin, W., & Dou, Y. (2020) · Journal of Systems Science and Systems Engineering · doi:10.1007/s11518-020-5473-1 · Cited by 15 · see citing works
Addressing distributional shifts in operations management: The case of order fulfillment in customized production
Senoner, J., Kratzwald, B., Kuzmanovic, M., Netland, T. H., & Feuerriegel, S. (2023) · Production and Operations Management · doi:10.1111/poms.14021 · Cited by 13 · see citing works
Design and Evaluation of Personalized Free TrialsPreprint
Yoganarasimhan, H., Barzegary, E., & Pani, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3616641 · Cited by 13 · see citing works
Multiobjective Personalization of Marketing Interventions
Rafieian, O., Kapoor, A., & Sharma, A. (2024) · Marketing Science · doi:10.1287/mksc.2023.0122 · Cited by 12 · see citing works
A Commentary on the Application of Artificial Intelligence in the Insurance Industry
Singh, S. K. (2020) · Trends in Artificial Intelligence · doi:10.36959/643/305 · Cited by 12 · see citing works
Competitive Model Selection in Algorithmic Targeting
Iyer, G. V., & Ke, T. T. (2024) · Marketing Science · doi:10.1287/mksc.2023.0175 · Cited by 11 · see citing works
Detection of Financial Opportunities in Micro-Blogging Data With a Stacked Classification System
de Arriba-Pérez, F., García-Méndez, S., Regueiro-Janeiro, J. A., & González-Castaño, F. J. (2020) · IEEE Access · doi:10.1109/access.2020.3041084 · Cited by 11 · see citing works
Bridging the Gap: A Systematic Benchmarking of Uplift Modeling and Heterogeneous Treatment Effects Methods
Rößler, J., & Schoder, D. (2022) · Journal of Interactive Marketing · doi:10.1177/10949968221111083 · Cited by 11 · see citing works
Improving Imbalanced Machine Learning with Neighborhood-Informed Synthetic Sample Placement
Nasir, M., Dağ, A., Simsek, S., Ivanov, A., & Oztekin, A. (2022) · Journal of Management Information Systems · doi:10.1080/07421222.2022.2127453 · Cited by 11 · see citing works
Doing More with Less: Overcoming Ineffective Long-Term Targeting Using Short-Term Signals
Huang, T.-W., & Ascarza, E. (2024) · Marketing Science · doi:10.1287/mksc.2022.0379 · Cited by 11 · see citing works
Optimal Targeting in Fundraising: A Machine-Learning ApproachPreprint
Cagala, T., Glogowsky, U., Rincke, J., & Strittmatter, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3836338 · Cited by 10 · see citing works
Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical ApproachPreprint
Wang, Y., Tao, L., & Zhang, X. X. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4602954 · Cited by 10 · see citing works
Targeted Automation of Order Decisions Using Machine LearningPreprint
Imdahl, C., Hoberg, K., & Schmidt, W. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3822131 · Cited by 9 · see citing works
Applying Large Language Models to Sponsored Search Advertising
Reisenbichler, M., Reutterer, T., & Schweidel, D. A. (2025) · Marketing Science · doi:10.1287/mksc.2023.0611 · Cited by 8 · see citing works
Training Demand Prediction Models by Decision Error for Two-Stage Lot-Sizing Problems
Gong, H., Zhang, Y., & Zhang, Z. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3248623 · Cited by 8 · see citing works
EXPLORING THE CONFLUENCE OF BIG DATA, ARTIFICIAL INTELLIGENCE, AND DIGITAL MARKETING ANALYTICS: A COMPREHENSIVE REVIEW
Mahi, R., Alam, F., & Hasan, M. (2024) · GLOBAL MAINSTREAM JOURNAL · doi:10.62304/jieet.v3i3.159 · Cited by 22 · see citing works
Amazon's Artificial Intelligence in Retail Novelty - Case Study
Manasa, R., & Devi, A. J. (2022) · International Journal of Case Studies in Business IT and Education · doi:10.47992/ijcsbe.2581.6942.0233 · Cited by 11 · see citing works
The Design and Targeting of Compliance Promotions
Daljord, Ø., Mela, C. F., Roos, J. M. T., Sprigg, J., & Yao, S. (2023) · Marketing Science · doi:10.1287/mksc.2022.1420 · Cited by 8 · see citing works
Forecasting the nearly unforecastable: why aren’t airline bookings adhering to the prediction algorithm?
Thirumuruganathan, S., Jung, S., Robillos, D. R., Salminen, J. O., & Jansen, B. J. J. (2021) · Electronic Commerce Research · doi:10.1007/s10660-021-09457-0 · Cited by 8 · see citing works
Advancement of human-machine collaboration in manufacturing: A review on industry 1.0 - 6.0
Oladele, I. O., Okoro, C. J., Falana, S. O., Olanrewaju, O. F., Adelani, S. O., & Onuh, L. N. (2024) · JOURNAL OF MECHANICAL ENGINEERING AND SCIENCES · doi:10.15282/jmes.18.4.2024.7.0813 · Cited by 8 · see citing works
Customer Acquisition via Explainable Deep Reinforcement Learning
Song, Y., Wang, W., & Yao, S. (2024) · Information Systems Research · doi:10.1287/isre.2022.0529 · Cited by 7 · see citing works
Artifical Intelligence and CRM
Dixit, S. (2021) · Advances in marketing, customer relationship management, and e-services book series · doi:10.4018/978-1-7998-7959-6.ch006 · Cited by 9 · see citing works
Targeting, Personalization, and Engagement in an Agricultural Advisory ServicePreprint
Athey, S. C., Cole, S. A., Nath, S., & Zhu, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4536641 · Cited by 7 · see citing works
Predictive Analytics in Marketing Using Artificial Intelligence
Shukla, R. P., Juneja, D., & Monga, S. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-9531-8_17 · Cited by 5 · see citing works
Prediction and Optimization for Multi-Product Marketing Resource Allocation in Cross-Border E-Commerce
Xie, Y., Ye, H.-Q., & Zhu, W. (2025) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer20020124 · Cited by 5 · see citing works
A Sample Size Calculation for Training and Certifying Targeting Policies
Simester, D. I., Timoshenko, A., & Zoumpoulis, S. I. (2022) · Management Science · doi:10.1287/mnsc.2022.02947 · Cited by 6 · see citing works
Self-citation
A Consensus Algorithm for Linear Support Vector Machines
Dutta, H. (2021) · Management Science · doi:10.1287/mnsc.2021.4042 · Cited by 5 · see citing works
Incrementality Representation Learning: Synergizing Past Experiments for Intervention PersonalizationPreprint
Huang, T.-W., Ascarza, E., & Israeli, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4859809 · Cited by 5 · see citing works
Predicting Postoperative Complications in Cancer Patients: A Survey Bridging Classical and Machine Learning Contributions to Postsurgical Risk Analysis
Gonçalves, D. M., Henriques, R., & Costa, R. S. (2021) · Cancers · doi:10.3390/cancers13133217 · Cited by 4 · see citing works
Neural Optimization with Adaptive Heuristics for Intelligent Marketing System
Wei, C., Zelditch, B., Chen, J., Ribeiro, A. A. S. T., Tay, J. K., Ocejo, B., Selvaraj, S. K., Gupta, A., & Almeida, L. B. D. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671591 · Cited by 4 · see citing works
Artificial Intelligence/Machine Learning Driven Decision making in Business Analytics for Financial Sector using Ensemble Machine Learning Techniques
Agrawal, R., Desai, S. S., Dholwani, D., Kedari, N., & Banerjee, A. (2024) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic61668.2024.10731028 · Cited by 4 · see citing works
Techniques for Evaluating the Robustness of Deep Learning Systems: A Preliminary Review
França, H. L., Teixeira, C., & Laranjeiro, N. (2021) · Latin-American Symposium on Dependable Computing · doi:10.1109/ladc53747.2021.9672592 · Cited by 4 · see citing works
The Future of Artificial Intelligence in Southeast Asia
Mongkol, K. (2022) · Advances in finance, accounting, and economics book series · doi:10.4018/978-1-6684-5849-5.ch002 · Cited by 4 · see citing works
Machine Learning Based Decision-Making: A Sensemaking Perspective
Li, J. (., Namvar, M., Im, G. P., & Akhlaghpour, S. (2024) · AJIS. Australasian journal of information systems/AJIS. Australian journal of information systems/Australian journal of information systems · doi:10.3127/ajis.v28.4781 · Cited by 9 · see citing works
Balancing Risk and Profit: Predicting the Performance of Potential New Customers in the Insurance Industry
Soriano-Gonzalez, R., Tsertsvadze, V., Osorio, C., Fuster, N., Juan, A. A., & Bernabeu, E. P. (2024) · Information · doi:10.3390/info15090546 · Cited by 4 · see citing works
Optimal Price TargetingPreprint
Smith, A. N., Seiler, S., & Aggarwal, I. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3822459 · Cited by 4 · see citing works
How Do Content Producers Respond to Engagement on Social Media Platforms?Preprint
Mummalaneni, S., Yoganarasimhan, H., & Pathak, V. V. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4173537 · Cited by 4 · see citing works
Empowering a Sustainable Future: Fostering Sustainable Behavior with Targeted InterventionsPreprint
Krefeld-Schwalb, A., & Gabel, S. (2023) · doi:10.31234/osf.io/nc2bh · Cited by 4 · see citing works
Improving Targeting with Differentially-Private Data: Honest Covariate-Adaptive Calibration of Treatment EffectsPreprint
Huang, T.-W., & Ascarza, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4575240 · Cited by 4 · see citing works
Do Crowds Validate False Data? Systematic Distortion and Affective Polarization
Pienta, D., Somanchi, S., Vishwamitra, N., Berente, N., & Thatcher, J. B. (2024) · MIS Quarterly · doi:10.25300/misq/2024/17482 · Cited by 3 · see citing works
How Do Content Producers Respond to Engagement on Social Media Platforms?
Mummalaneni, S., Yoganarasimhan, H., & Pathak, V. V. (2026) · Marketing Science · doi:10.1287/mksc.2023.0178 · Cited by 3 · see citing works
Artificial Intelligence/Machine Learning driven decision making in business analytics for Financial Sector using Ensemble Machine Learning Techniques
Agrawal, R., Desai, S. S., Dholwani, D., Kedari, N., & Banerjee, A. (2024) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic61668.2024.10730914 · Cited by 3 · see citing works
Optimizing Scalable Targeted Marketing Policies with Constraints
Lu, H., Simester, D. I., & Zhu, Y. (2025) · Marketing Science · doi:10.1287/mksc.2023.0640 · Cited by 3 · see citing works
Self-citation
Understanding Consumption Reduction Through the TPB: Moderating Effects of the Need for Evaluation and Self-Referencing Individual Differences
Nejad, S. G., Skeiseid, H. V., & Derdowski, L. A. (2024) · Journal of Intelligence · doi:10.3390/jintelligence12110119 · Cited by 5 · see citing works
Towards Accountability in Machine Learning Applications: A System-testing ApproachPreprint
Wan, W. X., & Lindenthal, T. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3758451 · Cited by 3 · see citing works
Competitive Model Selection in Algorithmic Targeting
Iyer, G. V., & Ke, T. T. (2023) · National Bureau of Economic Research · doi:10.3386/w31002 · Cited by 3 · see citing works
Optimizing Scalable Targeted Marketing Policies with ConstraintsPreprint
Lu, H., Simester, D. I., & Zhu, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4668582 · Cited by 3 · see citing works
Self-citation
Making Meaningful User Segments from Datasets Using Product Dissemination and Product Impact
Jansen, B. J. J., Salminen, J. O., & Jung, S. (2020) · Data and Information Management · doi:10.2478/dim-2020-0048 · Cited by 2 · see citing works
Pazarlamada Yapay Zekâ
Aktaş, M., & Çavuşoğlu, S. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub398.c1662 · Cited by 3 · see citing works
Revealed Wisdom of the Crowd: Bids Predict Loan Quality
Yao, J., Lin, M., & Wu, D. J. (2025) · Management Science · doi:10.1287/mnsc.2022.02575 · Cited by 2 · see citing works
The Design and Targeting of Compliance PromotionsPreprint
Daljord, Ø., Mela, C. F., Roos, J. M. T., Sprigg, J., & Yao, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4006675 · Cited by 2 · see citing works
Customer Acquisition Via Explainable Deep Reinforcement LearningPreprint
Song, Y., Wang, W., & Yao, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4802411 · Cited by 2 · see citing works
The Market Basket Transformer: A New Foundation Model for RetailPreprint
Gabel, S., & Ringel, D. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4335141 · Cited by 2 · see citing works
Policy-Aware Experimentation: Strategic Sampling for Optimized Targeting PoliciesPreprint
Chen, Y.-W., Ascarza, E., & Netzer, O. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5044549 · Cited by 2 · see citing works
An Empirical Investigation of the Demographic Differences in Attitude Toward Femvertising: The Case of Turkey
Yurcu, M. E. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub398.c1667 · Cited by 1 · see citing works
Pazarlama Araştırmalarında Yapay Zeka (AI) ile Müşteri Deneyimi: Tüketici Davranışlarının Analizinde Yapay Zeka Çözümleri
Atlı, H. F. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub398.c1663 · Cited by 1 · see citing works
Deneyimsel Pazarlama Kavramının Bibliyometrik Analiz ile İncelenmesi
Erdem, A. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub398.c1665 · Cited by 1 · see citing works
Revolutionizing Marketing through Artificial Intelligence: An Analytical Compendium of Case Studies
Agarwal, S., Agarwal, S., Mittal, V., & Anuradha (2024) · International Conference on Intelligent Systems and Advanced Applications (ICISAA) · doi:10.1109/icisaa62385.2024.10828777 · Cited by 1 · see citing works
A Deep-DiD Method to Estimate Heterogeneous Treatment Effects: Application to Content Creator Selection
Cheng, Y., Wang, J., Cao, X., Shen, Z. M., & Zhang, Y. (2026) · Marketing Science · doi:10.1287/mksc.2023.0511 · Cited by 1 · see citing works
A decision support framework for estimating the impact of covariate shift in machine learning systems
Meire, M., Hoornaert, S., Caigny, A. D., & Coussement, K. (2026) · Decision Support Systems · doi:10.1016/j.dss.2026.114632 · Cited by 1 · see citing works
From Human Hands to Machine Minds: Financing AI ‐Driven Entrepreneurship in Reward‐Based Crowdfunding
Song, Z., Lan, Y., & Shan, Y. G. (2026) · Accounting and Finance · doi:10.1111/acfi.70207 · Cited by 2 · see citing works
Tedarik Zinciri Bakış Açısıyla Şirketlerin Yeşil Aklama ile Başa Çıkma Stratejileri
Sofyalıoğlu, Ç., & Kartal, B. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub398.c1661 · Cited by 1 · see citing works
Model-Free Approximate Bayesian Learning for Large-Scale Conversion Funnel Optimization
Iyengar, G., & Singal, R. (2024) · Production and Operations Management · doi:10.1177/10591478241231857 · Cited by 1 · see citing works
To do or not to do: cost-sensitive causal decision-makingPreprint
Olaya, D., Verbeke, W., Belle, J. V., & Guerry, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01407 · Cited by 1 · see citing works
Sustained Learning Under Algorithm-Driven AutomationPreprint
Imdahl, C., Schmidt, W., & Hoberg, K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4292438 · Cited by 1 · see citing works
Multi-Objective Personalization of the Length and Skippability of Video AdvertisementsPreprint
Rafieian, O., Kapoor, A., & Sharma, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4394969 · Cited by 2 · see citing works

Efficiently Evaluating Targeting Policies Using Field ExperimentsPreprint
Simester, D. I., Timoshenko, A., & Zoumpoulis, S. I. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3017384 · Cited by 3 · see citing works
Self-citation
Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments
Simester, D. I., Timoshenko, A., & Zoumpoulis, S. I. (2020) · Management Science · doi:10.1287/mnsc.2019.3379 · Cited by 54 · see citing works
Self-citation
Leveraging Deep-learning and Field Experiment Response Heterogeneity to Enhance Customer Targeting Effectiveness
Zhang, K., & Luo, X. (2019) · Journal of the Association for Information Systems · Cited by 4 · see citing works
Improving Targeting Policies Using Transfer LearningPreprint
Ibragimov, M., Simester, D. I., & Timoshenko, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146292 · Cited by 1 · see citing works
Self-citation
Machine Learning Algorithms in Target Marketing Analysis
Wu, J. (2024) · Highlights in Science Engineering and Technology · doi:10.54097/v8zrvc27 · Cited by 2 · see citing works

Authors: Simester, D., Timoshenko, A., & Zoumpoulis, S. I.
Affiliations: Massachusetts Institute of Technology (US)
INSEAD (FR)
Corresponding author: Simester, Duncan
ORCID: Simester, Duncan
Timoshenko, Artem
Zoumpoulis, Spyros I.
Venue: Management Science
Publisher: Institute for Operations Research and the Management Sciences (INFORMS)
Published: 2020-06
Volume: 66
Issue: 6
Pages: 2495-2522
ISSN: 0025-1909 (ISSN-L)
1526-5501
DOI: 10.1287/mnsc.2019.3308
OpenAlex: W2985046451
Cited by: 133
References: 68
Citations by year: 2026: 13 · 2025: 14 · 2024: 30 · 2023: 30 · 2022: 19 · 2021: 19 · 2020: 7 · 2018: 1
FWCI: 14.76
Type: journal article
Language: en
Open access: green
hdl.handle.net
Licence: cc-by-nc-sa
Locations: Management Science (publishedVersion)
DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
Topics: Social Sciences › Business, Management and Accounting › Marketing › Customer churn and segmentation
Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
Keywords: customer prospecting, field experiments, covariate shift, concept shift, data aggregation, class imbalance, classification methods, training data quality, new customer acquisition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Targeting Prospective Customers: Robustness of Machine-Learning Methods to Typical Data Challenges

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks