A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
This article reviews current methods for evaluating models in the cognitive sciences, including theoretically based approaches, such as Bayes factors and minimum description length measures; simulation approaches, including model mimicry evaluations; and practical approaches, such as validation and generalization measures. This article argues that, although often useful in specific settings, most of these approaches are limited in their ability to give a general assessment of models. This article argues that hierarchical methods, generally, and hierarchical Bayesian methods, specifically, can provide a more thorough evaluation of models in the cognitive sciences. This article presents two worked examples of hierarchical Bayesian analyses to demonstrate how the approach addresses key questions of descriptive adequacy, parameter interference, prediction, and generalization in principled and coherent ways.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558
Cross validatory choice and assessment of statistical predictions
Stone, M. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00994.x · Cited by 10,881
Markov Chain Monte Carlo in Practice
(1995) · Chapman & Hall/CRC eBooks · doi:10.1201/b14835 · Cited by 7,757
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,362
Insensitivity to future consequences following damage to human prefrontal cortex
Bechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (1994) · Cognition · doi:10.1016/0010-0277(94)90018-3 · Cited by 6,128
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
Cited by
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E.-J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,627
Self-citationHDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python
Wiecki, T. V., Sofer, I., & Frank, M. J. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00014 · Cited by 1,047
“Under the assumption that participants within each group are similar to each other, but not identical, a hierarchical model can be constructed where individual parameter estimates are constrained by group-level distributions (Shiffrin et al., 2008; Nilsson et al., 2011).”
BackgroundA simple introduction to Markov Chain Monte–Carlo sampling
van Ravenzwaaij, D., Cassey, P., & Brown, S. D. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1015-8 · Cited by 621
Jones, M., & Love, B. C. (2011) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10003134 · Cited by 533
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Matzke, D., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.5.798 · Cited by 461
Self-citationBenchmarks for models of short-term and working memory.
Oberauer, K., Lewandowsky, S., Awh, E., Brown, G. D. A., Conway, A., Cowan, N., Donkin, C., Farrell, S., Hitch, G. J., Hurlstone, M. J., et al. (2018) · Psychological Bulletin · doi:10.1037/bul0000153 · Cited by 447
Bayesian data analysis for newcomers
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1272-1 · Cited by 399
Ahn, W.-Y., Haines, N., & Zhang, L. (2017) · Computational Psychiatry · doi:10.1162/cpsy_a_00002 · Cited by 362
How cognitive modeling can benefit from hierarchical Bayesian models
Lee, M. D. (2011) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.013 · Cited by 307
Self-citationA tutorial on approximate Bayesian computation
Turner, B. M., & Van Zandt, T. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.02.005 · Cited by 293
Gronau, Q. F., Sarafoglou, A., Matzke, D., Ly, A., Boehm, U., Marsman, M., Leslie, D. S., Forster, J. J., Wagenmakers, E.-J., & Steingroever, H. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2017.09.005 · Cited by 284
Self-citationStatistical methods for linguistic research: Foundational Ideas—Part II
Nicenboim, B., & Vasishth, S. (2016) · Language and Linguistics Compass · doi:10.1111/lnc3.12207 · Cited by 263
Toward a principled Bayesian workflow in cognitive science.
Schad, D. J., Betancourt, M., & Vasishth, S. (2021) · Psychological Methods · doi:10.1037/met0000275 · Cited by 252
Time perception: the bad news and the good
Matthews, W. J., & Meck, W. H. (2014) · WIREs Cognitive Science · doi:10.1002/wcs.1298 · Cited by 179
Heathcote, A., Lin, Y.-S., Reynolds, A., Strickland, L., Gretton, M., & Matzke, D. (2019) · Behavior Research Methods · doi:10.3758/s13428-018-1067-y · Cited by 168
A model-based fMRI analysis with hierarchical Bayesian parameter estimation.
Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R., & Brown, J. W. (2011) · Journal of Neuroscience, Psychology, and Economics · doi:10.1037/a0020684 · Cited by 162
Self-citationWetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143
Self-citationScholl, J., Kolling, N., Nelissen, N., Wittmann, M. K., Harmer, C. J., & Rushworth, M. F. S. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.0396-15.2015 · Cited by 85
Interregional alpha-band synchrony supports temporal cross-modal integration
van Driel, J., Knapen, T., van Es, D. M., & Cohen, M. X. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.07.022 · Cited by 66
“Psychophysical evidence of cross-modal integration versus distractor interference Using a hierarchical Bayesian modeling approach (Shiffrin et al., 2008) in fitting a psychometric function on the behavioral data of each subject, we found strong evidence for the auditory modality being superior over…”
MethodFrom bad to worse: Symbolic equivalence and opposition in fear generalisation
Bennett, M., Hermans, D., Dymond, S., Vervoort, E., & Baeyens, F. (2015) · Cognition and Emotion · doi:10.1080/02699931.2014.973833 · Cited by 27
Parsimonious estimation of signal detection models from confidence ratings
Selker, R., van den Bergh, D., Criss, A. H., & Wagenmakers, E.-J. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01231-3 · Cited by 18
Self-citation
Metadata
- Authors
- Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.
- Venue
- Cognitive Science
- Publisher
- Wiley
- Published
- 2008-12
- Volume
- 32
- Issue
- 8
- Pages
- 1248-1284
- ISSN
- 0364-0213 (ISSN-L)
- 1551-6709
- OpenAlex
- W2148412534
- Cited by
- 348
- References
- 70
- FWCI
- 15.52
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.