The DOI matches its Crossref record; 116 of the 117 references it cites were found in a registry or an index.
Model Uncertainty, Data Mining and Statistical Inference
Abstract
This paper takes a broad, pragmatic view of statistical inference to include all aspects of model formulation. The estimation of model parameters traditionally assumes that a model has a prespecified known form and takes no account of possible uncertainty regarding the model structure. This implicitly assumes the existence of a 'true' model, which many would regard as a fiction. In practice model uncertainty is a fact of life and likely to be more serious than other sources of uncertainty which have received far more attention from statisticians. This is true whether the model is specified on subject-matter grounds or, as is increasingly the case, when a model is formulated, fitted and checked on the same data set in an iterative, interactive way. Modern computing power allows a large number of models to be considered and data-dependent specification searches have become the norm in many areas of statistics. The term data mining may be used in this context when the analyst goes to great lengths to obtain a good fit. This paper reviews the effects of model uncertainty, such as too narrow prediction intervals, and the non-trivial biases in parameter estimates which can follow data-based modelling. Ways of assessing and overcoming the effects of model uncertainty are discussed, including the use of simulation and resampling methods, a Bayesian model averaging approach and collecting additional data wherever possible. Perhaps the main aim of the paper is to ensure that statisticians are aware of the problems and start addressing the issues even if there is no simple, general theoretical fix.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“in reality, the true model is unknown, may not have a generalized linear form, or may not even exist [22].”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [22]
Citations
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Book Review: Time Series Analysis: Forecasting and Control
Geurts, M. D., Box, G. E. P., & Jenkins, G. M. (1977) · Journal of Marketing Research · doi:10.2307/3150485 · Cited by 19,410
Statistical Methods for Research Workers
Fisher, R. A. (1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,883
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765
The Advanced Theory of Statistics.
Clarke, G. M., Kendall, M. G., & Stuart, A. (1978) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344782 · Cited by 6,727
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229
Box, G. E. P. (1976) · Journal of the American Statistical Association · doi:10.1080/01621459.1976.10480949 · Cited by 2,114
Bayesian Forecasting and Dynamic Models
West, M., & Harrison, J. (1997) · Springer eBooks · doi:10.1007/b98971 · Cited by 1,946
Cross-Validation of Regression Models
Picard, R. R., & Cook, R. D. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478083 · Cited by 1,551
Combining forecasts: A review and annotated bibliography
(1990) · Long Range Planning · doi:10.1016/0024-6301(90)90316-v · Cited by 1,508
Cited by
Sorted by citations
Items 901–1,000 of 1,000
A blooming business : Identifying limits to Lake Taihu's nutrient input
Janssen, A. B. G. (2017) · doi:10.18174/413490 · Cited by 1
Assessing Bayesian Nonparametric Log-Linear Models: an application to Disclosure Risk estimationPreprint
Carota, C., Filippone, M., & Polettini, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05244 · Cited by 1
Cichosz, P. (2015) · doi:10.1002/9781118950951.ch15 · Cited by 1
Sacral Pressure Injury Study Commentary
Kaylor, J., Bankieris, K., Johnson, A., & Hooper, V. D. (2019) · AORN Journal · doi:10.1002/aorn.12823 · Cited by 1
An Assessment of Bayesian Model-Averaged Logistic Regression for Intensive-Care PrognosisPreprint
Dybowski, R. (2020) · medRxiv · doi:10.1101/2020.06.08.20124412 · Cited by 1
Monitoring process and assessing uncertainty for ANFIS time series forecasting
Deng, Y. (2002) · doi:10.33915/etd.2437 · Cited by 1
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_9 · Cited by 1
Green, J. W., Springer, T. A., & Holbech, H. (2018) · doi:10.1002/9781119488798.refs · Cited by 1
Forecast Intervals for Infectious Disease ModelsPreprint
Picard, R., & Osthus, D. (2022) · medRxiv · doi:10.1101/2022.04.29.22274494 · Cited by 1
Discussion of the Paper by Goldstein and Spiegelhalter
(1996) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.1996.tb00766.x · Cited by 1
The Structure of Simultaneous Equations Estimators
Hendry, D. F. (2000) · doi:10.1093/0198293542.003.0014 · Cited by 1
Discussion of the Paper by Draper
(1995) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1995.tb02016.x · Cited by 1
Pairwise Collinearity Detection Using Parallel Algorithms: Preliminary DetailsPreprint
Tîrnȃucȃ, C., & Homocianu, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4742523 · Cited by 1
Estimation After Model Building: A First Step
Miller, A. J. (1996) · COMPSTAT · doi:10.1007/978-3-642-46992-3_48 · Cited by 1
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_15
Efficient Subject-Oriented Evaluating and Mining Methods for Data with Schema Uncertainty
Wang, Y., Tang, C., Wang, T., Yang, D., & Zhu, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25853-4_25
35 Bridging the biostatistics-epidemiology gap
Edwards, L. J. (2000) · Handbook of statistics · doi:10.1016/s0169-7161(00)18037-x
Multivariate and Multilevel Longitudinal Analysis
Longford, N. T. (2010) · Springer eBooks · doi:10.1007/978-3-642-11760-2_3
Wheat yield prediction modeling for localized optimization of fertilizer and herbicide application
Wagner, N. C. (2004) · Montana State University ScholarWorks (Montana State University)
Fournier, N. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5075/epfl-thesis-5043
Why doesn\'t your model pass information to mine?
Giles, J. R. A., & Kessler, H. (2009)
Diagnostic and prognostic models: applications and methods
Zuithoff, N. P. A. (2012) · Utrecht University Repository (Utrecht University)
Fasteignamat með tímaleiðréttu margvíðu aðhvarfslíkani
Helgason, J. Á. (2011)
Preliminaries for Nonparametric Regression
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_10
DATA MINING AND STATISTICS: EXAMINING CRITICAL PATTERNS OF RESEARCH AND PRACTICE
Wang, A. (2007) · UTA ResearchCommons (University of Texas Arlington)
Parameter and Model Uncertainty
Cairns, A. J. G. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat04467
Hennebry, R. (2011) · SETU Waterford Libraries - Open Access Repository
Recursive Thick Modeling and the Choice of Monetary Policy in MexicoPreprint
Rodríguez, A., & Rodríguez, P. N. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.933729
Three Essays in Environmental and Agricultural Issues
Sengupta, S. (2010) · doi:10.31274/etd-180810-1786
Comparison of Two Gas Selection Methodologies: An Application of Bayesian Model Averaging
Renholds, A. S., Thompson, S. E., Anderson, K. K., & Chilton, L. K. (2006) · doi:10.2172/1023740
Combining generalized linear models
Chen, L. (2005) · doi:10.31274/rtd-180813-16444
Good to Great, or Great Data Mining?
Beck, K. L., & Niendorf, B. D. (2016)
Hybrid data regression modelling in measurementPreprint
Bókov, V. B. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0808.4031
Computer-simulation data inclusive regression modelling in measurement
Bókov, V. B. (2009) · Accreditation and Quality Assurance · doi:10.1007/s00769-009-0532-9
Predicting Prostate Cancer Behavior—What is a Urologist to Do?
Kattan, M. W. (2008) · The Journal of Urology · doi:10.1016/j.juro.2008.02.002
Sistema de Apoio à Decisão para a Classificação de Risco em Projectos de Tecnologias de Informação
Ramalho, T. J. M. F. (2011) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Functions of Normal Random Variables
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_17
SMALL AREA ESTIMATION OF UNEMPLOYMENT FOR SOUTH AFRICAN LABOUR MARKET STATISTICS
Hakizimana, J.-M. (2012) · University of the Witwatersrand, Johannesburg Institutional Repository on DSpace (University of the Witwatersrand, Johannesburg)
Avoiding biases from data-dependent specification search: an application to a tillage choice modelPreprint
Sengupta, S., Kurkalova, L. A., & Kling, C. L. (2006) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.21399
On the appropriateness of inappropriate VaR models
Härdle, W. K., Hlávka, Z., & Stahl, G. (2006) · Allgemeines Statistisches Archiv · doi:10.1007/s10182-006-0234-0
Graphical methods of determining predictor importance and effect
Rendahl, A. K. (2008) · University of Minnesota Digital Conservancy (University of Minnesota)
A Parametricness Index and Consistency with Complexity Penalty for Model Selection
Liu, W. (2010) · University of Minnesota Digital Conservancy (University of Minnesota)
Optimization under uncertainty: Sensitivity analysis and regret
Tan, C. H. (2012)
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_6
Wouters, K., Abrahantes, J. C., Molenberghs, G., Geys, H., Ahnaou, A., Drinkenburg, W. H. I. M., & Bijnens, L. (2010) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543401003618744
Prediction models in growth research using advanced statistical methods
de Ridder, M. A. J. (2008) · EUR Research Repository (Erasmus University Rotterdam)
Khoorani, A., Farajzadeh, M., Bazgeer, S., & Abbass, B. (2014)
Analysis of oxygenation and other risk factors of retinopathy of prematurity in preterm babies
Zahari, M. (2015) · UC Research Repository (University of Canterbury) · doi:10.26021/2214
Wakefield, J. (2012) · Springer series in statistics · doi:10.1007/978-1-4419-0925-1_19
Medeiros, L. D. C., & Bressan, A. A. (2015) · Brazilian Business Review · doi:10.15728/bbr.2015.12.3.4
Forecasting the West Texas Intermediate Crude Oil Spot Price Using Multi Model InferencePreprint
Ryan, L., & Emery, M. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2079341
Short term ozone effects on morbidity for the city of Milano, Italy, 1996-2003
Chiogna, M., & Pauli, F. (2008) · Padua@research (University of Padova)
Wann sind falsche VaR-Modelle dennoch ad
Hl, Z. (2003)
Data Mining Tools for Exploring Big Data
Stine, R. A. (2014)
Data mining tasks and methods: Equation fitting: multidimensional regression analysis
Rao, J. S., & Potts, W. J. (2002) · Oxford University Press eBooks
Macroeconometric modelling: Alternative approaches
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0002
Impulse response and trend/cycle properties of the UK model
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0010
Probability forecasting: Concepts and analysis
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0007
Smith, G. A. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0010
Global modelling and other applications
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0012
Probability event forecasting with the UK model
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0011
Selection of Classification Models Using Data Envelopment Analysis
Nakhaeizadeh, G., & Taylor, C. C. (1998) · Studies in classification, data analysis, and knowledge organization · doi:10.1007/978-3-642-72253-0_83
An economic theory of the long run
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0004
An economic theory of the short run
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0005
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0008
Garratt, A., Lee, K. C., Pesaran, M. H., & Shin, Y. (2006) · doi:10.1093/0199296855.003.0006
MODELE PROGNOZ EKONOMETRYCZNYCH
Zmarzłowski, K., & Karwański, M. (2015)
Forecasting Swap Spreads:A Bayesian Approach
Klein, D., Nikitina, E., & Curtillet, J.-C. (2016) · The Journal of Fixed Income · doi:10.3905/jfi.2016.26.2.040
Tree models: a Bayesian perspective
Egan, B. (2006) · Queen Mary Research Online (Queen Mary University of London)
Darby, P. (2016) · Loughborough University Institutional Repository (Loughborough University)
Modelling Challenges for Climate and Community Resilient Socioecological Systems
Dey, A., Gupta, A. K., & Singh, G. (2016) · Springer eBooks · doi:10.1007/978-3-319-43901-3_11
Fissha, S., & Berhanu, M. (2018) · Journal of Public Administration and Governance · doi:10.5296/jpag.v8i2.13279
Bayesian Inference in Classification and Discrimination
Press, S. J. (2002) · Wiley series in probability and statistics · doi:10.1002/9780470317105.ch16
Optimizing pharmacokinetic studies utilizing microsampling
Barnett, H. (2017) · Lancaster EPrints (Lancaster University) · doi:10.17635/lancaster/thesis/185
Variable Selection Deviation Measures
Nan, Y. (2015) · University of Minnesota Digital Conservancy (University of Minnesota)
Flexible Generalized Mixture Model Cluster Analysis Withelliptically-Contoured Distributions
(2014) · i-manager’s Journal on Pattern Recognition · doi:10.26634/jpr.1.1.2820
Brady, H. E., Nicosia, N., & Seto, E. Y. (2002) · eScholarship (California Digital Library)
On the Appropriateness of Inappropriate VAR ModelsPreprint
Härdle, W. K., Hlávka, Z., & Stahl, G. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2894388
dos Santos, C. B., Pedroso, B., Guimarães, A. M., Pìlatti, L. A., & Kovaleski, J. L. (2019) · Dialnet (Universidad de la Rioja)
Matthews, J. N. S. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a09030
Regression and CausalityPreprint
Schomaker, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11754 · Cited by 4
Frequency Domain Model Selection for Servo Systems ensuring Practical Identifiability
Tantau, M., Wielitzka, M., Ortmaier, T., & Perner, L. (2020) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma49215.2020.9233569
If You Torture the Data Long Enough
Smith, G. E. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0008
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0012
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0014
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0007
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0009
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0011
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0003
Smith, G. (2018) · Oxford University Press eBooks · doi:10.1093/oso/9780198824305.003.0005
The Curse of Explanation: Model Selection in Language Testing Research
Choi, I. (2020) · Springer eBooks · doi:10.1007/978-981-15-8952-2_9
Kwiatkowska, M., Urban, J., & Dagasso, G. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.05.035
Testing Theories with Qualitative and Quantitative PredictionsPreprint
Coleman, S. R. (2021) · doi:10.31235/osf.io/d8g26
Model selection for servo control systems
Wielitzka, M., Perner, L., & Tantau, M. (2021) · International Journal of Mechatronics and Automation · doi:10.1504/ijma.2021.10038414
Hahn, S., Williamson, P. R., Hutton, J. L., Garner, P., & Flynn, E. V. (2000) · Statistics in Medicine · doi:10.1002/1097-0258(20001230)19:24
Walter (1998) · doi:10.1002/(sici)1097-0258(19980115)17:1
Metadata
- Authors
- Chatfield, C.
- Affiliations
- University of Bath (GB)
- Corresponding author
- Chatfield, Chris
- Venue
- Journal of the Royal Statistical Society. Series A (Statistics in Society)
- Publisher
- Oxford University Press (OUP)
- Published
- 1995
- Volume
- 158
- Issue
- 3
- Pages
- 419
- ISSN
- 0964-1998 (ISSN-L)
- OpenAlex
- W2060050130
- Cited by
- 1,103
- References
- 117
- Citations by year
- 2026: 18 · 2025: 34 · 2024: 38 · 2023: 33 · 2022: 35 · 2021: 33 · 2020: 44 · 2019: 28 · 2018: 47 · 2017: 35 · 2016: 37 · 2015: 51 · 2014: 52 · 2013: 28 · 2012: 63
- FWCI
- 24.46
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- model uncertainty, data mining, statistical inference, model selection, prediction intervals, estimation bias, Bayesian model averaging, resampling, simulation, model misspecification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
