CoChat
Sign Up
Cited by 1,810Markov Chains and Monte Carlo MethodsStatistical Methods and Bayesian InferenceData Analysis with R

Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion)

Open access
Vehtari, Aki, Gelman, Andrew, Simpson, Daniel, Carpenter, Bob, Bürkner, Paul-Christian·2021·Bayesian Analysis
DOI 10.1214/20-ba1221
Open this paper in CoChat — free accountOpen PDF

Abstract

Markov chain Monte Carlo is a key computational tool in Bayesian statistics, but it can be challenging to monitor the convergence of an iterative stochastic algorithm. In this paper we show that the convergence diagnostic Rˆ of Gelman and Rubin (1992) has serious flaws. Traditional Rˆ will fail to correctly diagnose convergence failures when the chain has a heavy tail or when the variance varies across the chains. In this paper we propose an alternative rank-based diagnostic that fixes these problems. We also introduce a collection of quantile-based local efficiency measures, along with a practical approach for computing Monte Carlo error estimates for quantiles. We suggest that common trace plots should be replaced with rank plots from multiple chains. Finally, we give recommendations for how these methods should be used in practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1214/20-ba1221

Cites

45 of 47 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • All model estimates met the convergence criteria of R-hat less than 1.01 and effective sample size greater than 1000.

    “All of our model estimates met these criteria.”

    From The Effects of Generative AI on Design Fixation and Divergent Thinking (2024) · cites this paper as [65]

    In “Recommendations for practice” (page unavailable), the source recommends R-hat below 1.01, but its recommended rank-normalized ESS threshold is greater than 400, not 1000. The citing authors’ statement that their own estimates met the criteria is their result; this source supports only part of the criteria they invoke. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Model inference was performed by Hamiltonian Monte Carlo with the NUTS sampler using PyStan, and model convergence was ensured, with the potential scale reduction factor satisfying R̂ ≤ 1.02 for all model parameters and the smallest effective sample size for all model parameters greater than 500.

    “Model inference was performed by Hamiltonian Monte Carlo with the NUTS sampler using PyStan 69 , the Python implementation of Stan.”

    From Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA (2021) · cites this paper as [71]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 24

  1. Blom, G. (1958). Statistical Estimates and Transformed Beta-Variables. Wiley; New York. MR0095553. 678
  2. Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017). "Stan: A Probabilistic Pro- gramming Language." Journal of Statistical Software, Articles, 76(1): 1-32. 668
  3. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis, third edition. CRC Press. MR3235677. 668, 669, 670, 672, 673, 675, 676, 687
  4. Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. R. (2003). Bayesian Data Analysis, second edition. Chapman & Hall. MR2027492. 675
  5. Lunn, D. J., Thomas, A., Best, N., and Spiegelhalter, D. (2000). "WinBUGS- a Bayesian modelling framework: Concepts, structure, and extensibility." Statistics and Computing, 10(4): 325-337. 668
  6. Mengersen, K. L., Robert, C. P., and Guihenneuc-Jouyaux, C. (1999). "MCMC con- vergence diagnostics: A review." In Bernardo, J. M., Berger, J. O., and Dawid, A. P. (eds.), Bayesian Statistics 6 , 415-440. Oxford University Press. MR1723507. 668, 674, 675
  7. Vats, D. and Knudson, C. (2018). "Revisiting the Gelman-Rubin Diagnostic." arXiv:1812.09384. 671, 672, 673, 674, 676 · arXiv
  8. Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., and Bürkner, P.-C. (2020). "Rank- Normalization, Folding, and Localization: An Improved R for Assessing Convergence of MCMC. Supplementary Material." Bayesian Analysis. doi: https://doi.org/10. 1214/20-BA1221SUPP. 669 · DOI
  9. Anderson, T. W. (1971). The Statistical Analysis of Time Series. New York: John Wiley & Son. MR0283939. 700
  10. Qin, Q. and Hobert, J. P. (2021). "On the limitations of single-step drift and minoriza- tion in Markov chain convergence analysis." To appear in Annals of Applied Proba- bility. 699
  11. Robertson, N., Flegal, J. M., Vats, D., and Jones, G. L. (2021). "Assessing and visu- alizing simultaneous simulation error." To appear in Journal of Computational and Graphical Statistics. 698
  12. Rosenthal, J. S. (1995). "Minorization conditions and convergence rates for Markov chain Monte Carlo." Journal of the American Statistical Association, 90: 558-566. MR1340509. 699
  13. Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M. A., Guo, J., Li, P., and Riddell, A. (2017). "Stan: A probabilistic pro- gramming language." Journal of Statistical Software, 76(1): 1-32. 703
  14. Cook, R. D. (1994). "On the interpretation of regression plots." Journal of the American Statistical Association, 89: 177-189. MR1266295. 705
  15. de Valpine, P., Turek, D., Paciorek, C. J., Anderson-Bergman, C., Lang, D. T., and Bodik, R. (2017). "Programming with models: Writing statistical algorithms for general model structures with NIMBLE." Journal of Computational and Graphi- cal Statistics, 26: 403-413. MR3640196. doi: https://doi.org/10.1080/10618600. 2016.1172487. 703 · DOI
  16. Gelfand, A. E. and Smith, A. F. M. (1990). "Sampling-based approaches to calculating marginal densities." Journal of the American Statistical Association, 85: 398-409. MR1141740. 708
  17. Gelman, A. and Rubin, D. B. (1992b). "A single sequence from the Gibbs sampler gives a false sense of security." In Bernardo, J. M., Berger, J. O., Dawid, A. P., and Smith, A. F. M. (eds.), Bayesian Statistics 4 , 625-631. Oxford: Oxford University Press. MR1723490. 703
  18. Lunn, D., Spiegelhalter, D., Thomas, A., and Best, N. (2009). "The BUGS project: Evolution, critique and future directions." Statistics in Medicine, 28: 3049-3067. MR2750401. doi: https://doi.org/10.1002/sim.3680. 703 · DOI · Paper page
  19. Plummer, M. (2003). "JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling." In Proceedings of the 3rd International Workshop on Distributed Statistical Computing, volume 124, 1-10. 703
  20. Brooks, S. P. B. and Gelman, A. G. (1998). "General methods for monitoring conver- gence of iterative simulations." Journal of Computational and Graphical Statistics, 7(4): 434-455. MR1665662. doi: https://doi.org/10.2307/1390675. 718 · DOI
  21. Dhaka, A. K., Catalina, A., Andersen, M. R., Magnusson, M., Huggins, J., and Ve- htari, A. (2020). "Robust, Accurate Stochastic Optimization for Variational Infer- ence." In Advances in Neural Information Processing Systems, volume 33, 10961- 10973. 718
  22. Lambert, B. and Vehtari, A. (2021). "R*: A Robust MCMC Convergence Diagnostic with Uncertainty Using Decision Tree Classifiers." Bayesian Analysis, 1(1): 1-27. 718
  23. Säilynoja, T., Bürkner, P.-C., and Vehtari, A. (2021). "Graphical Test for Discrete Uniformity and its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison." arXiv preprint arXiv:2103.10522. 713 · arXiv
  24. Vehtari, A., Simpson, D., Gelman, A., Yuling, Y., and Gabry, J. (2019). "Pareto smoothed importance sampling." arXiv preprint arXiv:1507.02646. 715 · arXiv

Cited by

Sorted by citations

Related

Authors
Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C.
Affiliations
  • Aalto University (FI)
  • Columbia University (US)
  • University of Toronto (CA)
  • Flatiron Institute (US)
ORCID
Venue
Bayesian Analysis
Publisher
Institute of Mathematical Statistics
Published
2021-06-01
Volume
16
Issue
2
Pages
667-718
ISSN
  • 1931-6690 (ISSN-L)
  • 1936-0975
DOI
10.1214/20-ba1221
OpenAlex
W2920804790
Cited by
1,810
References
32
Citations by year
2026: 499 · 2025: 424 · 2024: 328 · 2023: 263 · 2022: 153 · 2021: 93 · 2020: 37 · 2019: 11
FWCI
96.56
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Bayesian Analysis (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Data Analysis with R
Keywords
convergence diagnostics, Markov chain Monte Carlo, heavy-tailed distributions, Monte Carlo error estimation, trace plots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review