Testing the Validity of a Demand Model: An Operations Perspective
Open accessAbstract
The fields of statistics and econometrics have developed powerful methods for testing the validity (specification) of a model based on its fit to underlying data. Unlike statisticians, managers are typically more interested in the performance of a decision rather than the statistical validity of the underlying model. We propose a framework and a statistical test that incorporate decision performance into a measure of statistical validity. Under general conditions on the objective function, asymptotic behavior of our test admits a sharp and simple characterization. We develop our approach in a revenue management setting and apply the test to a data set used to optimize prices for consumer loans. We show that traditional model-based goodness-of-fit tests may consistently reject simple parametric models of consumer response (e.g., the ubiquitous logit model), while at the same time these models may “pass” the proposed performance-based test. Such situations arise when decisions derived from a postulated (and possibly incorrect) model generate results that cannot be distinguished statistically from the best achievable performance—i.e., when demand relationships are fully known.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many optimization projects do not materialize because performance is measured exclusively based on forecast rather than realized profits.
“many optimization projects do not materialize because the performance is measured exclusively based on the forecast rather than the realized profits (a similar observation motivated the work by Besbes et al. 2010).”
From Clearance Pricing Optimization for a Fast-Fashion Retailer (2012) · cites this paper as Besbes et al. 2010
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Maximum Likelihood Estimation of Misspecified Models
White, H. (1982) · Econometrica · doi:10.2307/1912526 · Cited by 4,175 · see citing works
Applied Nonparametric Regression
Härdle, W. K. (1990) · Cambridge University Press eBooks · doi:10.1017/ccol0521382483 · Cited by 2,487 · see citing works
The Theory and Practice of Revenue Management
Talluri, K. T., & Van Ryzin, G. J. (2004) · International Series in Operations Research & Management Science · doi:10.1007/b139000 · Cited by 2,243 · see citing works
Comparing Nonparametric Versus Parametric Regression Fits
Härdle, W. K., & Mammen, E. (1993) · The Annals of Statistics · doi:10.1214/aos/1176349403 · Cited by 1,204 · see citing works
Pagan, A., & Ullah, A. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9780511612503 · Cited by 953 · see citing works
Statistical Analysis of a Telephone Call Center
Brown, L., Gans, N., Mandelbaum, A., Sakov, A., Shen, H., Zeltyn, S., & Zhao, L. H. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001808 · Cited by 783 · see citing works
Pricing and Revenue Optimization
Phillips, R. L. (2005) · Stanford University Press eBooks · doi:10.1515/9780804781640 · Cited by 621 · see citing works
Self-citationBuilding Models for Marketing Decisions
Leeflang, P. S. H., Wittink, D. R., Wedel, M., & Naert, P. A. (2000) · International series in quantitative marketing · doi:10.1007/978-1-4615-4050-2 · Cited by 503 · see citing works
Berger, J. O. (1980) · Springer series in statistics · doi:10.1007/978-1-4757-1727-3 · Cited by 452 · see citing works
A Consistent Conditional Moment Test of Functional Form
Bierens, H. J. (1990) · Econometrica · doi:10.2307/2938323 · Cited by 445 · see citing works
Bootstrapping General Empirical Measures
Evarist Gine, M., & Zinn, J. (1990) · The Annals of Probability · doi:10.1214/aop/1176990862 · Cited by 383 · see citing works
Andrews, D. W. K. (1997) · Econometrica · doi:10.2307/2171880 · Cited by 255 · see citing works
A practical inventory control policy using operational statistics
Liyanage, L. H., & Shanthikumar, J. G. (2004) · Operations Research Letters · doi:10.1016/j.orl.2004.08.003 · Cited by 216 · see citing works
Models of the Spiral-Down Effect in Revenue Management
Cooper, W. L., Homem‐de‐Mello, T., & Kleywegt, A. J. (2006) · Operations Research · doi:10.1287/opre.1060.0304 · Cited by 185 · see citing works
Semiparametric estimation of a work-trip mode choice model
Horowitz, J. L. (1993) · Journal of Econometrics · doi:10.1016/0304-4076(93)90113-j · Cited by 132 · see citing works
Cachon, G. P., & Kök, A. G. (2007) · Manufacturing & Service Operations Management · doi:10.1287/msom.1060.0145 · Cited by 130 · see citing works
Operations related groups (ORGs): A clustering procedure for production/inventory systems
Ernst, R., & Cohen, M. A. (1990) · Journal of Operations Management · doi:10.1016/0272-6963(90)90010-b · Cited by 101 · see citing works
A Nonparametric Density Estimation Method for Brand Choice Using Scanner Data
Abe, M. (1995) · Marketing Science · doi:10.1287/mksc.14.3.300 · Cited by 31 · see citing works
On nonparametric kernel estimation of the mode of the regression function in the random design model
Ziegler, K. (2002) · Journal of nonparametric statistics · doi:10.1080/10485250215321 · Cited by 25 · see citing works
The Use and Value of Models in Decision Analysis
Nickerson, R. C., & Boyd, D. W. (1980) · Operations Research · doi:10.1287/opre.28.1.139 · Cited by 23 · see citing works
Testing the Multinomial Logit Model
Bartels, K., Boztuğ, Y., & Müller, M. (2000) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-642-57280-7_32 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Amemiya, T. (1981), 'Qualitative response models: A survey', Journal of Economic Literature XIX, 1483-1536.
- Ben-Akiva, M. and Lerman, S. R. (1985), Discrete Choice Analysis: Theory and Application to Travel Demand, MIT Press.
- Train, K. (2002), Discrete Choice Methods with Simulation, Cambridge University Press.
- White, H. (1996), Estimation, Inference and Specification Analysis, Cambridge University Press.
Cited by
Sorted by citations
Smart “Predict, then Optimize”
Elmachtoub, A. N., & Grigas, P. (2021) · Management Science · doi:10.1287/mnsc.2020.3922 · Cited by 878 · see citing works
On the (Surprising) Sufficiency of Linear Models for Dynamic Pricing with Demand Learning
Besbes, O., & Zeevi, A. J. (2015) · Management Science · doi:10.1287/mnsc.2014.2031 · Cited by 225 · see citing works
Self-citationClearance Pricing Optimization for a Fast-Fashion Retailer
Caro, F., & Gallien, J. (2012) · Operations Research · doi:10.1287/opre.1120.1102 · Cited by 203 · see citing works
Multi‐Methodological Research in Operations Management
Choi, T., Cheng, T. C. E., & Zhao, X. (2016) · Production and Operations Management · doi:10.1111/poms.12534 · Cited by 206 · see citing works
Markdown Pricing with Unknown Fraction of Strategic Customers
Mersereau, A. J., & Zhang, D. (2012) · Manufacturing & Service Operations Management · doi:10.1287/msom.1120.0376 · Cited by 106 · see citing works
A Comparative Empirical Study of Discrete Choice Models in Retail Operations
Berbeglia, G., Garassino, A., & Vulcano, G. J. (2018) · Management Science · doi:10.1287/mnsc.2021.4069 · Cited by 104 · see citing works
The Value of Personalized Pricing
Elmachtoub, A. N., Gupta, V., & Hamilton, M. (2018) · Management Science · doi:10.1287/mnsc.2020.3821 · Cited by 98 · see citing works
Optimization of Tree Ensembles
Mišić, V. V. (2020) · Operations Research · doi:10.1287/opre.2019.1928 · Cited by 94 · see citing works
He, B., Dexter, F., Macario, A., & Zenios, S. A. (2011) · Manufacturing & Service Operations Management · doi:10.1287/msom.1110.0350 · Cited by 91 · see citing works
On the Minimax Complexity of Pricing in a Changing Environment
Besbes, O., & Zeevi, A. J. (2011) · Operations Research · doi:10.1287/opre.1100.0867 · Cited by 84 · see citing works
Self-citationLearning and Pricing with Models That Do Not Explicitly Incorporate Competition
Cooper, W. L., Homem‐de‐Mello, T., & Kleywegt, A. J. (2015) · Operations Research · doi:10.1287/opre.2014.1341 · Cited by 79 · see citing works
Choosing arrival process models for service systems: Tests of a nonhomogeneous Poisson process
Kim, S., & Whitt, W. (2014) · Naval Research Logistics (NRL) · doi:10.1002/nav.21568 · Cited by 50 · see citing works
Kao, Y.-H., Roy, B. V., & Yan, X. (2009) · Cited by 36 · see citing works
The role of optimization in some recent advances in data-driven decision-making
Baardman, L., Cristian, R., Perakis, G., Singhvi, D., Lami, O. S., & Thayaparan, L. (2022) · Mathematical Programming · doi:10.1007/s10107-022-01874-9 · Cited by 30 · see citing works
Constrained optimization of objective functions determined from random forests
Biggs, M., Hariss, R., & Perakis, G. (2022) · Production and Operations Management · doi:10.1111/poms.13877 · Cited by 29 · see citing works
Dynamic Valuation of Delinquent Credit-Card Accounts
Chehrazi, N., & Weber, T. A. (2011) · Management Science · doi:10.1287/mnsc.2015.2203 · Cited by 29 · see citing works
Optimizing Objective Functions Determined from Random ForestsPreprint
Biggs, M., & Hariss, R. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2986630 · Cited by 28 · see citing works
Newsvendor-type models with decision-dependent uncertainty
Lee, S., Homem‐de‐Mello, T., & Kleywegt, A. J. (2012) · Mathematical Methods of Operations Research · doi:10.1007/s00186-012-0396-3 · Cited by 25 · see citing works
Erkip, N. K. (2022) · European Journal of Operational Research · doi:10.1016/j.ejor.2022.08.024 · Cited by 25 · see citing works
Monotone Approximation of Decision Problems
Chehrazi, N., & Weber, T. A. (2010) · Operations Research · doi:10.1287/opre.1100.0814 · Cited by 24 · see citing works
Now Playing: DVD Purchasing for a Multilocation Rental Firm
Baron, O., Hajizadeh, I., & Milner, J. M. (2010) · Manufacturing & Service Operations Management · doi:10.1287/msom.1100.0314 · Cited by 23 · see citing works
Mind the gap between research and practice in operations management
Chen, X., Deng, T., Shen, Z. M., & Yu, Y. F. (2022) · IISE Transactions · doi:10.1080/24725854.2022.2046893 · Cited by 15 · see citing works
Risk-Based Loan Pricing: Portfolio Optimization Approach with Marginal Risk Contribution
Chun, S. Y., & Lejeune, M. A. (2016) · Management Science · doi:10.1287/mnsc.2019.3378 · Cited by 15 · see citing works
Data-driven inventory management under customer substitution
Müller, S., Huber, J., Bubak, A., Fleischmann, M., & Stuckenschmidt, H. (2025) · European Journal of Operational Research · doi:10.1016/j.ejor.2025.12.034 · Cited by 15 · see citing works
Decision-based model selection
den Boer, A. V., & Sierag, D. (2020) · European Journal of Operational Research · doi:10.1016/j.ejor.2020.08.025 · Cited by 14 · see citing works
Clearance Pricing Optimization for a Fast-Fashion Retailer
Caro, F., & Gallien, J. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1731402 · Cited by 203 · see citing works
On the Use of Buy Up as a Model of Customer Choice in Revenue Management
Cooper, W. L., & Li, L. (2012) · Production and Operations Management · doi:10.1111/j.1937-5956.2012.01352.x · Cited by 13 · see citing works
Heteroscedastic Exponomial Choice
Alptekinoğlu, A., & Semple, J. H. (2018) · Operations Research · doi:10.1287/opre.2020.2074 · Cited by 13 · see citing works
The Power and Limits of Predictive Approaches to Observational-Data-Driven OptimizationPreprint
Bertsimas, D., & Kallus, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.02347 · Cited by 19 · see citing works
Directed Principal Component Analysis
Kao, Y.-H., & Roy, B. V. (2014) · Operations Research · doi:10.1287/opre.2014.1290 · Cited by 11 · see citing works
Arevalillo, J. M. (2020) · Information Sciences · doi:10.1016/j.ins.2020.12.039 · Cited by 10 · see citing works
Study of demand models and price optimization performance
Lee, S. (2011) · SMARTech Repository (Georgia Institute of Technology) · Cited by 8 · see citing works
Multichannel Advertising: Budget Allocation in the Presence of Spillover and Carryover Effects
Chen, H., Chen, Y., Park, S.-H., & Shin, D. (2025) · Manufacturing & Service Operations Management · doi:10.1287/msom.2023.0293 · Cited by 7 · see citing works
Bertsimas, D., & Kallus, N. (2022) · INFORMS Journal on Optimization · doi:10.1287/ijoo.2022.0077 · Cited by 7 · see citing works
Profit-based unit commitment models with price-responsive decision-dependent uncertainty
Lejeune, M. A., Dehghanian, P., & Ma, W. (2023) · European Journal of Operational Research · doi:10.1016/j.ejor.2023.12.006 · Cited by 6 · see citing works
Coordination of Inventory Distribution and Price Markdowns for Clearance Sales at Zara
Caro, F., Babio, F., & Peña, F. (2019) · Springer series in supply chain management · doi:10.1007/978-3-030-20119-7_13 · Cited by 5 · see citing works
Pricing from Observational DataPreprint
Bertsimas, D., & Kallus, N. (2016) · arXiv (Cornell University) · Cited by 4 · see citing works
Proxy-Aided Demand Learning with an Application to Various Pricing Problems
Shen, T., & Cui, Y. (2025) · Operations Research · doi:10.1287/opre.2025.1793 · Cited by 2 · see citing works
Gallego, G., & Topaloğlu, H. (2019) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-1-4939-9606-3_8 · Cited by 2 · see citing works
Fernández-Loría, C., & Provost, F. J. (2022) · INFORMS Journal on Data Science · doi:10.1287/ijds.2022.0013 · Cited by 2 · see citing works
Observational vs. Experimental Data When Making Automated Decisions Using Machine Learning
Fernández-Loría, C., & Provost, F. J. (2025) · INFORMS Journal on Data Science · doi:10.1287/ijds.2023.0012 · Cited by 3 · see citing works
Cross‐estimation for decision selection
Gu, X., & Li, B. (2020) · Applied Stochastic Models in Business and Industry · doi:10.1002/asmb.2542 · Cited by 1 · see citing works
Observational vs Experimental Data When Making Automated Decisions Using Machine LearningPreprint
Fernández-Loría, C., & Provost, F. J. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3444678 · Cited by 1 · see citing works
Pricing Tools and Analysis for Emerging e-Commerce Technologies
Hamilton, M. (2019) · doi:10.7916/d8-sxa6-q384
On Data-Driven Multi-Product Pricing
Wang, T., Wu, C., & Qi, W. M. (2020) · IEEE Control Systems Letters · doi:10.1109/lcsys.2020.3043591
Evaluating Predictive Models in the Orienteering Problem with Stochastic Profits: A Simulation Study
Çakır, F. (2021) · Journal of Business Research - Turk · doi:10.20491/isarder.2021.1184
On Data-driven Multi-Product Pricing
Wang, T., Wu, C., & Qi, W. M. (2021) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc50511.2021.9482649
Enabling Collaborative Diagnosis through Novel Distributed Learning System with AutonomyPreprint
Liu, X., Wang, Y., Tang, S., & Lin, S.-B. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4128032
- Authors
- Besbes, O., Phillips, R., & Zeevi, A.
- Affiliations
- Columbia University (US)
- Venue
- Manufacturing & Service Operations Management
- Published
- 2010-01
- Volume
- 12
- Issue
- 1
- Pages
- 162-183
- ISSN
- 1523-4614 (ISSN-L)
- 1526-5498
- OpenAlex
- W2135094465
- Cited by
- 47
- References
- 23
- Citations by year
- 2025: 4 · 2023: 1 · 2022: 7 · 2021: 3 · 2020: 5 · 2019: 4 · 2018: 3 · 2017: 1 · 2016: 3 · 2015: 3 · 2014: 2 · 2012: 4
- FWCI
- 6.89
- Type
- journal article
- Language
- en
- Open access
- green
- repository.upenn.edu
- Locations
- Manufacturing & Service Operations Management (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- Topics
- Social Sciences › Business, Management and Accounting › Management Information Systems › Supply Chain and Inventory Management
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Business, Management and Accounting › Marketing › Consumer Market Behavior and Pricing
- Keywords
- specification testing, demand models, decision performance, revenue management, logit model, goodness-of-fit tests, asymptotic behavior, consumer response, price optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
