Cited by 1,103Statistical Methods and InferenceStatistical Methods and Bayesian InferenceStatistical Methods and Applications

Model uncertainty, data mining and statistical inference

Chatfield, Chris·1995·Journal of the Royal Statistical Society. Series A (Statistics in Society)

Abstract

This paper takes a broad, pragmatic view of statistical inference to include all aspects of model formulation. The estimation of model parameters traditionally assumes that a model has a prespecified known form and takes no account of possible uncertainty regarding the model structure. This implicitly assumes the existence of a 'true' model, which many would regard as a fiction. In practice model uncertainty is a fact of life and likely to be more serious than other sources of uncertainty which have received far more attention from statisticians. This is true whether the model is specified on subject-matter grounds or, as is increasingly the case, when a model is formulated, fitted and checked on the same data set in an iterative, interactive way. Modern computing power allows a large number of models to be considered and data-dependent specification searches have become the norm in many areas of statistics. The term data mining may be used in this context when the analyst goes to great lengths to obtain a good fit. This paper reviews the effects of model uncertainty, such as too narrow prediction intervals, and the non-trivial biases in parameter estimates which can follow data-based modelling. Ways of assessing and overcoming the effects of model uncertainty are discussed, including the use of simulation and resampling methods, a Bayesian model averaging approach and collecting additional data wherever possible. Perhaps the main aim of the paper is to ensure that statisticians are aware of the problems and start addressing the issues even if there is no simple, general theoretical fix.

Integrity signals

Registry

Crossref · 10.2307/2983440

Cites

116 of 116 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

1 of the 117 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Chatfield, C.
Affiliations
University of Bath (GB)
Corresponding author
Chatfield, Chris
Venue
Journal of the Royal Statistical Society. Series A (Statistics in Society)
Published
1995
Volume
158
Issue
3
Pages
419
ISSN
0964-1998 (ISSN-L)
OpenAlex
W2060050130
Cited by
1,103
References
117
Citations by year
2026: 18 · 2025: 34 · 2024: 38 · 2023: 33 · 2022: 35 · 2021: 33 · 2020: 44 · 2019: 28 · 2018: 47 · 2017: 35 · 2016: 37 · 2015: 51 · 2014: 52 · 2013: 28 · 2012: 63
FWCI
24.46
Type
journal article
Language
en
Open access
closed
Locations
Journal of the Royal Statistical Society Series A (Statistics in Society) (publishedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
Keywords
model uncertainty, data mining, statistical inference, model selection, prediction intervals, estimation bias, Bayesian model averaging, resampling, simulation, model misspecification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review