The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Toward a principled Bayesian workflow in cognitive science.
Open accessAbstract
Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan, and easily accessible front-end packages such as brms. The utility of Bayesian methods, however, ultimately depends on the relevance of the Bayesian model, in particular whether or not it accurately captures the structure of the data and the data analyst's domain expertise. Even with powerful software, the analyst is responsible for verifying the utility of their model. To demonstrate this point, we introduce a principled Bayesian workflow (Betancourt, 2018) to cognitive science. Using a concrete working example, we describe basic questions one should ask about the model: prior predictive checks, computational faithfulness, model sensitivity, and posterior predictive checks. The running example for demonstrating the workflow is data on reading times with a linguistic manipulation of object versus subject relative clause sentences. This principled Bayesian workflow also demonstrates how to use domain knowledge to inform prior distributions. It provides guidelines and checks for valid data analysis, avoiding overfitting complex models to noise, and capturing relevant data structure in a probabilistic model. Given the increasing use of Bayesian methods, we aim to discuss how these methods can be properly employed to obtain robust answers to scientific questions. All data and code accompanying this article are available from https://osf.io/b2vx9/. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Appendix S3 in the online Supporting Information presents the prior predictive checks (Schad, Betancourt, & Vasishth, 2021).”
From Effects of Availability, Contingency, and Formulaicity on the Accuracy of English Grammatical Morphemes in Second Language Writing (2022) · cites this paper as Schad, Betancourt, & Vasishth, 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We followed a Bayesian workflow for both computational models and conventional analyses [19, 20].”
From Predicting emotional valence in autism: a preregistered study in the Bayesian Brain framework (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328
brms : An R Package for Bayesian Multilevel Models Using Stan
Bürkner, P.-C. (2017) · Journal of Statistical Software · doi:10.18637/jss.v080.i01 · Cited by 9,476 · cited 7×
“p in the literature. It assumes that readers have had some previous exposure to and experience with basic concepts of Bayesian modeling, such as fitting simple Bayesian models (e.g., in brms or Stan) (Bürkner, 2017a; Kruschke, 2014; Vasishth et al., 2018b) , and are interested in how to put together their knowledge abo…”
BackgroundMethodInfluentialMixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540
Stan : A Probabilistic Programming Language
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017) · Journal of Statistical Software · doi:10.18637/jss.v076.i01 · Cited by 7,754 · cited 16×
“Much progress has been made in the development of probabilistic programming languages such as Stan (Carpenter et al., 2017), WinBUGS (Lunn, Thomas, Best, & Spiegelhalter, 2000), JASP (JASP Team, 2019), and JAGS (Plummer, 2012).”
BackgroundMethodInfluentialSelf-citationInformation theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,536
(2015) · doi:10.1007/s11222-016-9696-4 · Cited by 5,611
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. J. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,397
Kmenta, J. (1997) · University of Michigan Press eBooks · doi:10.3998/mpub.15701 · Cited by 3,240
Advanced Bayesian Multilevel Modeling with the R Package brms
Bürkner, P.-C. (2018) · The R Journal · doi:10.32614/rj-2018-017 · Cited by 3,015
Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2021) · Bayesian Analysis · doi:10.1214/20-ba1221 · Cited by 1,810
(2014) · Thammasat University Digital Collections · Cited by 1,732
Robustness in the Strategy of Scientific Model Building
Box, G. E. P. (1979) · Elsevier eBooks · doi:10.1016/b978-0-12-438150-6.50018-2 · Cited by 1,698
(2013) · Cited by 1,351
Generating random correlation matrices based on vines and extended onion method
Lewandowski, D., Kurowicka, D., & Joe, H. (2009) · Journal of Multivariate Analysis · doi:10.1016/j.jmva.2009.04.008 · Cited by 1,135
Visualization in Bayesian Workflow
Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., & Gelman, A. (2019) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12378 · Cited by 1,122
Self-citationLindley, D. V. (1957) · Biometrika · doi:10.1093/biomet/44.1-2.187 · Cited by 994
Bayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 810
Probability and the Weighing of Evidence.
M., P. A., & Good, I. J. (1950) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981046 · Cited by 753
The Deviance Information Criterion: 12 Years on
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2014) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/rssb.12062 · Cited by 631
Validating Bayesian Inference Algorithms with Simulation-Based CalibrationPreprint
Talts, S., Betancourt, M., Simpson, D., Vehtari, A., & Gelman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.06788 · Cited by 506
Self-citationThe Prior Can Often Only Be Understood in the Context of the Likelihood
Gelman, A., Simpson, D., & Betancourt, M. (2017) · Entropy · doi:10.7916/d8v99mjx · Cited by 419
Self-citationLy, A., Verhagen, J., & Wagenmakers, E. (2015) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.06.004 · Cited by 410
Link, W. A., & Eaton, M. J. (2012) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2011.00131.x · Cited by 351
A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods
Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E. (2008) · Cognitive Science · doi:10.1080/03640210802414826 · Cited by 346
Validation of Software for Bayesian Models Using Posterior Quantiles
Cook, S. R., Gelman, A., & Rubin, D. B. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x136976 · Cited by 345
Bayesian Benefits for the Pragmatic Researcher
Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016) · Current Directions in Psychological Science · doi:10.1177/0963721416643289 · Cited by 334
The philosophy of Bayes factors and the quantification of statistical evidence
Morey, R. D., Romeijn, J., & Rouder, J. N. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.001 · Cited by 324
How cognitive modeling can benefit from hierarchical Bayesian models
Lee, M. D. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.013 · Cited by 313
Gronau, Q. F., Sarafoglou, A., Matzke, D., Ly, A., Boehm, U., Marsman, M., Leslie, D. S., Forster, J. J., Wagenmakers, E.-J., & Steingroever, H. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2017.09.005 · Cited by 284
Bayesian data analysis in the phonetic sciences: A tutorial introduction
Vasishth, S., Nicenboim, B., Beckman, M. E., Li, F., & Kong, E. J. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.07.008 · Cited by 246
Self-citationHierarchical Bayesian parameter estimation for cumulative prospect theory
Nilsson, H., Rieskamp, J., & Wagenmakers, E. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.006 · Cited by 240
Introduction to Bayesian Inference for Psychology
Etz, A. J., & Vandekerckhove, J. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1262-3 · Cited by 213
The statistical significance filter leads to overoptimistic expectations of replicability
Vasishth, S., Mertzen, D., Jäger, L. A., & Gelman, A. (2018) · Journal of Memory and Language · doi:10.1016/j.jml.2018.07.004 · Cited by 200
Self-citationProcessing Chinese relative clauses in context
Gibson, E., & Wu, H.-H. I. (2011) · Language and Cognitive Processes · doi:10.1080/01690965.2010.536656 · Cited by 183
Robust Modeling in Cognitive Science
Lee, M. D., Criss, A. H., Devezer, B., Donkin, C., Etz, A. J., Leite, F. P., Matzke, D., Rouder, J. N., Trueblood, J. S., White, C. N., et al. (2019) · Computational Brain & Behavior · doi:10.1007/s42113-019-00029-y · Cited by 148
Processing Chinese Relative Clauses: Evidence for the Subject-Relative Advantage
Vasishth, S., Chen, Z., Qiang, L., & Gueilan, G. (2013) · PLoS ONE · doi:10.1371/journal.pone.0077006 · Cited by 145
Self-citationHow to become a Bayesian in eight easy steps: An annotated reading list
Etz, A. J., Gronau, Q. F., Dablander, F., Edelsbrunner, P. A., & Baribault, B. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1317-5 · Cited by 120
Mulder, J., & Wagenmakers, E. (2016) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2016.01.002 · Cited by 107
Bayesian inference for psychology, part IV: parameter estimation and Bayes factors
Rouder, J. N., Haaf, J. M., & Vandekerckhove, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1420-7 · Cited by 102
Nicenboim, B., Vasishth, S., & Rösler, F. (2020) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2020.107427 · Cited by 101
Self-citationDeveloping constraint in bayesian mixed models.
Haaf, J. M., & Rouder, J. N. (2017) · Psychological Methods · doi:10.1037/met0000156 · Cited by 97
Nicenboim, B., & Vasishth, S. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2017.08.004 · Cited by 87
Self-citationHow to capitalize on a priori contrasts in linear (mixed) models: A tutorial
Schad, D. J., Vasishth, S., Hohenstein, S., & Kliegl, R. M. (2019) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104038 · Cited by 74
Self-citationA Bayesian hierarchical model for the measurement of working memory capacity
Morey, R. D. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.008 · Cited by 62
Ly, A., Boehm, U., Heathcote, A., Turner, B. M., Forstmann, B. U., Marsman, M., & Matzke, D. (2017) · doi:10.1002/9781119159193.ch34 · Cited by 51
Hierarchical single- and dual-process models of recognition memory
Pratte, M. S., & Rouder, J. N. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.007 · Cited by 45
Understanding memory impairment with memory models and hierarchical Bayesian analysis
Pooley, J. P., Lee, M., & Shankle, W. R. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.08.003 · Cited by 30
Bayesian hypothesis testing: Editorial to the Special Issue on Bayesian data analysis.
Hoijtink, H. J. A., & Chow, S. (2017) · Psychological Methods · doi:10.1037/met0000143 · Cited by 17
Does antecedent complexity affect ellipsis processing? An empirical investigation
Paape, D., Nicenboim, B., & Vasishth, S. (2017) · Glossa a journal of general linguistics · doi:10.5334/gjgl.290 · Cited by 6
Self-citation
Research works printed but not linked: 5
- Chow, S.-M., & Hoijtink, H. (2017). Bayesian estimation and modeling: Editorial to the second special issue on Bayesian data analysis. Psychological Methods, 22 (4), 609-615.
- Rabe, M. M., Vasishth, S., Hohenstein, S., Kliegl, R., & Schad, D. J. (2020). Hypr: An r package for hypothesis-driven contrast coding. PsyArXiv. Retrieved from 10.31234/osf.io/cqzdx · DOI
- Ratcliff, R., & Rouder, J. N. (2000). A diffusion model account of masking in two-choice letter identification. Journal of Experimental Psychology: Human Perception and Performance, 26 (1), 127.
- Vandekerckhove, J., Rouder, J. N., & Kruschke, J. K. (2018). Bayesian methods for advancing psychological science. Psychonomic Bulletin & Review, 25 (1), 1-4.
- Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27 (5), 1413-1432.
Cited by
Sorted by citations
Bayesian statistics and modelling
van de Schoot, R., Depaoli, S. A., King, R., Kramer, B. M. R., Märtens, K., Tadesse, M. G., Vannucci, M., Gelman, A., Veen, D., Willemsen, J. E., et al. (2021) · Nature Reviews Methods Primers · doi:10.1038/s43586-020-00001-2 · Cited by 1,338
Bayesian Analysis Reporting Guidelines
Kruschke, J. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01177-7 · Cited by 525
Addressing climate change with behavioral science: A global intervention tournament in 63 countries
Vlasceanu, M., Doell, K. C., Bak-Coleman, J. B., Todorova, B., Berkebile-Weinberg, M., Grayson, S., Patel, Y. H., Goldwert, D., Pei, Y., Chakroff, A., et al. (2024) · Science Advances · doi:10.1126/sciadv.adj5778 · Cited by 380
Interference patterns in subject-verb agreement and reflexives revisited: A large-sample study
Jäger, L. A., Mertzen, D., Van Dyke, J. A., & Vasishth, S. (2020) · Journal of Memory and Language · doi:10.1016/j.jml.2019.104063 · Cited by 142
Self-citationWorkflow techniques for the robust use of bayes factors.
Schad, D. J., Nicenboim, B., Bürkner, P., Betancourt, M., & Vasishth, S. (2022) · Psychological Methods · doi:10.1037/met0000472 · Cited by 134
Self-citation
Metadata
- Authors
- Schad, D. J., Betancourt, M., & Vasishth, S.
- Affiliations
- University of Potsdam (DE)
- Venue
- Psychological Methods
- Published
- 2021-02
- Volume
- 26
- Issue
- 1
- Pages
- 103-126
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2941987401
- Cited by
- 252
- References
- 51
- Citations by year
- 2026: 32 · 2025: 34 · 2024: 40 · 2023: 38 · 2022: 34 · 2021: 58
- FWCI
- 24.54
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- publish.UP (University of Potsdam) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (317633480—SFB 1287)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
- Keywords
- model checking, posterior predictive checks, model sensitivity, prior distributions, reading time, relative clauses, overfitting, probabilistic programming
- MeSH terms
- Bayes Theorem, Humans, Models, Psychological, Models, Statistical, Cognitive Science, Workflow
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
