How persuasive is a good fit? A comment on theory testing.

Abstract

Quantitative theories with free parameters often gain credence when they closely fit data. This is a mistake. A good fit reveals nothing about the flexibility of the theory (how much it cannot fit), the variability of the data (how firmly the data rule out what the theory cannot fit), or the likelihood of other outcomes (perhaps the theory could have fit any plausible result), and a reader needs all 3 pieces of information to decide how much the fit should increase belief in the theory. The use of good fits as evidence is not supported by philosophers of science nor by the history of psychology; there seem to be no examples of a theory supported mainly by good fits that has led to demonstrable progress. A better way to test a theory with free parameters is to determine how the theory constrains possible outcomes (i.e., what it predicts), assess how firmly actual outcomes agree with those constraints, and determine if plausible alternative outcomes would have been inconsistent with the theory, allowing for the variability of the data.

Cites (5)

1. A new look at the statistical model identification
   Akaike, H. · 1974 · IEEE Transactions on Automatic Control
2. The Methodology of Scientific Research Programmes: Philosophical Papers
   Lakatos, Imre · 1978
3. Stimulus information as a determinant of reaction time.
   Hyman, Ray · 1953 · Journal of Experimental Psychology
4. On the time relations of mental processes: An examination of systems of processes in cascade.
   McClelland, James L. · 1979 · Psychological Review
5. Overfitting Avoidance as Bias
   Schaffer, Cullen · 1993 · Machine Learning

Show all 5 references

Cited by (28)

1. Homo Heuristicus: Why Biased Minds Make Better Inferences
   Gigerenzer, Gerd, Brighton, Henry · 2009 · Topics in Cognitive Science
2. A Discounting Framework for Choice With Delayed and Probabilistic Rewards.
   Green, Leonard, Myerson, Joel · 2004 · Psychological Bulletin
3. Mindless statistics.
   Gigerenzer, Gerd · 2004 · The Journal of Socio-Economics
4. On the emancipation of PLS-SEM: a commentary on Rigdon (2012)
   Sarstedt, Marko, Ringle, Christian M., Henseler, Jörg, Hair, Joseph F. · 2014 · Long Range Planning
5. What mediation analysis can (not) do
   Fiedler, Klaus, Schott, Malte, Meiser, Thorsten · 2011 · Journal of Experimental Social Psychology

See all 875