The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches
Abstract
Researchers in comparative research increasingly use multilevel models to test effects of country‐level factors on individual behavior and preferences. However, the asymptotic justification of widely employed estimation strategies presumes large samples and applications in comparative politics routinely involve only a small number of countries. Thus, researchers and reviewers often wonder if these models are applicable at all. In other words, how many countries do we need for multilevel modeling? I present results from a large‐scale Monte Carlo experiment comparing the performance of multilevel models when few countries are available. I find that maximum likelihood estimates and confidence intervals can be severely biased, especially in models including cross‐level interactions. In contrast, the Bayesian approach proves to be far more robust and yields considerably more conservative tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Inference from iterative simulation using multiple sequences
Gelman, A., & Rubin, D. B. (1992) · Statistical Science · doi:10.1214/ss/1177011136 · Cited by 16,627
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,786
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Sampling-based approaches to calculating marginal densities
Gelfand, A. E., & Smith, A. F. M. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10476213 · Cited by 6,606
The EM Algorithm and Extensions
Kushary, D. (1998) · Technometrics · doi:10.1080/00401706.1998.10485534 · Cited by 5,417
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
(2006) · Technometrics · doi:10.1198/tech.2006.s417 · Cited by 4,950
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,575
Multilevel and Longitudinal Modeling Using Stata
(2006) · Biometrics · doi:10.1111/j.1541-0420.2006.00588_16.x · Cited by 4,236
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · cited 3×
“More sophisticated priors have been proposed (Natarajan and Kass 2000; Gelman 2006); however, I employ specifications which are commonly used by researchers (e.g. Spiegelhalter et al. 1997; Gelman and Hill 2007).”
BackgroundMethodInfluentialProbability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.86 · Cited by 3,582
Bayesian Analysis of Binary and Polychotomous Response Data
Albert, J. H., & Chib, S. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476321 · Cited by 3,167
Introducing Multilevel Modeling
Kreft, I., & de Leeuw, J. (1998) · doi:10.4135/9781849209366 · Cited by 2,997
Generalized, Linear, and Mixed Models
McCulloch, C. E., & Searle, S. R. (2000) · Wiley Series in Probability and Statistics · doi:10.1002/0471722073 · Cited by 2,994
Markov Chain Monte Carlo Convergence Diagnostics: A Comparative Review
Cowles, M. K., & Carlin, B. P. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476956 · Cited by 2,056
A Guide to Modern Econometrics
Verbeek, M. (2000) · RePEc: Research Papers in Economics · Cited by 1,767
Prior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,576
Goldstein, H. (2010) · Wiley Series in Probability and Statistics · doi:10.1002/9780470973394 · Cited by 1,568
The EM Algorithm and Extensions, 2E
McLachlan, G. J., & Krishnan, T. (2008) · Wiley Series in Probability and Statistics · doi:10.1002/9780470191613 · Cited by 1,529
Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models
Skrondal, A., & Rabe‐Hesketh, S. (2004) · Thammasat University Digital Collections · Cited by 1,505
Modeling Multilevel Data Structures
Steenbergen, M. R., & Jones, B. S. (2002) · American Journal of Political Science · doi:10.2307/3088424 · Cited by 1,503
Generalized Latent Variable Modeling
Skrondal, A., & Rabe-Hesketh, S. (2004) · Chapman & Hall/CRC eBooks · doi:10.1201/9780203489437 · Cited by 1,484
Multivariate Statistical Modelling Based on Generalized Linear Models
Fahrmeir, L., & Tutz, G. (1994) · Springer Series in Statistics · doi:10.1007/978-1-4899-0010-4 · Cited by 1,391
Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model
Pinheiro, J. C., & Bates, D. M. (1995) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1995.10474663 · Cited by 1,188
arm: Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Su, Y.-S. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.arm · Cited by 955
Robustness issues in multilevel regression analysis
Maas, C. J. M., & Hox, J. J. (2004) · Statistica Neerlandica · doi:10.1046/j.0039-0402.2003.00252.x · Cited by 837
Statistical Modelling Issues in School Effectiveness Studies
Aitkin, M., & Longford, N. (1986) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981882 · Cited by 790
Bayesian Analysis for the Social Sciences
Jackman, S. (2009) · Wiley Series in Probability and Statistics · doi:10.1002/9780470686621 · Cited by 756
Contextual Factors and the Extreme Right Vote in Western Europe, 1980–2002
Arzheimer, K. (2009) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00369.x · Cited by 750
Unifying Political Methodology: The Likelihood Theory of Statistical Inference
King, G. (1989) · Project Muse (Johns Hopkins University) · Cited by 710
The determinants of individual attitudes towards immigration
O'Rourke, K. H., & Sinnott, R. (2006) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2005.10.005 · Cited by 690
Gay Rights in the States: Public Opinion and Policy Responsiveness
LAX, J. R., & PHILLIPS, J. H. (2009) · American Political Science Review · doi:10.1017/s0003055409990050 · Cited by 666
Europeans and the European Community: the dynamics of public support for European integration
Eichenberg, R. C., & Dalton, R. J. (1993) · International Organization · doi:10.1017/s0020818300028083 · Cited by 655
Reliable Estimation of Generalized Linear Mixed Models using Adaptive Quadrature
Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2002) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0200200101 · Cited by 613
A comparison of Bayesian and likelihood-based methods for fitting multilevel models
Browne, W. J., & Draper, D. (2006) · Bayesian Analysis · doi:10.1214/06-ba117 · Cited by 603
Who Learns from What in Policy Diffusion Processes?
Gilardi, F. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2010.00452.x · Cited by 580
Bayesian Methods: A Social and Behavioral Sciences Approach
Gill, J. (2002) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420057478 · Cited by 579
Linear and Generalized Linear Mixed Models and Their Applications
(2007) · Springer Series in Statistics · doi:10.1007/978-0-387-47946-0 · Cited by 477
An Assessment of Estimation Procedures for Multilevel Models with Binary Responses
Rodriguez, G., & Goldman, N. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983404 · Cited by 476
Does Identity or Economic Rationality Drive Public Opinion on European Integration?
Hooghe, L., & Marks, G. (2004) · PS: Political Science & Politics · doi:10.1017/s1049096504004585 · Cited by 473
Iversen, T., & Rosenbluth, F. (2006) · American Journal of Political Science · doi:10.1111/j.1540-5907.2006.00166.x · Cited by 424
A simulation study of sample size for multilevel logistic regression models
Moineddin, R., Matheson, F. I., & Glazier, R. H. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-34 · Cited by 410
Weldon, S. A. (2006) · American Journal of Political Science · doi:10.1111/j.1540-5907.2006.00187.x · Cited by 401
Maas, C. J. M., & Hox, J. J. (2004) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2003.08.006 · Cited by 399
Economic Inequality and Intolerance: Attitudes toward Homosexuality in 35 Democracies
Andersen, R., & Fetner, T. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00352.x · Cited by 371
Ethnic Diversity and Generalized Trust in Europe: A Cross-National Multilevel Study
Hooghe, M., Reeskens, T., Stolle, D., & Trappers, A. (2009) · Comparative Political Studies · doi:10.1177/0010414008325286 · Cited by 362
The Insignificance of Null Hypothesis Significance Testing
Gill, J. (1999) · Political Research Quarterly · doi:10.1177/106591299905200309 · Cited by 338
Development, feasibility and performance of a health risk appraisal questionnaire for older persons
Stuck, A. E., Kharicha, K., Dapp, U., Anders, J., von Renteln-Kruse, W., Meier-Baumgartner, H. P., Harari, D., Swift, C. G., Ivanova, K., Egger, M., et al. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-1 · Cited by 324
Sufficient Sample Sizes for Multilevel Modeling
Maas, C. J. M., & Hox, J. J. (2005) · Methodology · doi:10.1027/1614-2241.1.3.85 · Cited by 302
Anderson, C. J., & Singer, M. M. (2008) · Comparative Political Studies · doi:10.1177/0010414007313113 · Cited by 286
Confidence Intervals vs Bayesian Intervals
Jaynes, E. T., & Kempthorne, O. (1976) · Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science · doi:10.1007/978-94-010-1436-6_6 · Cited by 285
Unifying Political Methodology: The Likelihood Theory of Statistical Inference
King, G. (1998) · University of Michigan Press eBooks · doi:10.3998/mpub.23784 · Cited by 278
The Impartiality of International Judges: Evidence from the European Court of Human Rights
VOETEN, E. (2008) · American Political Science Review · doi:10.1017/s0003055408080398 · Cited by 274
Bayesian Inference for Comparative Research
Western, B., & Jackman, S. (1994) · American Political Science Review · doi:10.2307/2944713 · Cited by 257
Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science
(1976) · Springer eBooks · doi:10.1007/978-94-010-1436-6 · Cited by 238
Landry, P. F., & Shen, M. (2005) · Political Analysis · doi:10.1093/pan/mpi001 · Cited by 229
Reference Bayesian Methods for Generalized Linear Mixed Models
Natarajan, R., & Kass, R. E. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473916 · Cited by 209
Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models
Browne, W. J., & Draper, D. (2000) · Computational Statistics · doi:10.1007/s001800000041 · Cited by 199
2. Random-Effects Modeling of Categorical Response Data
Agresti, A., Booth, J. G., Hobert, J. P., & Caffo, B. (2000) · Sociological Methodology · doi:10.1111/0081-1750.t01-1-00075 · Cited by 183
A Multilevel Analysis of School Examination Results
Goldstein, H., Rasbash, J., Yang, M., Woodhouse, G., Pan, H., Nuttall, D., & Thomas, S. (1993) · Oxford Review of Education · doi:10.1080/0305498930190401 · Cited by 182
Bayesian Methods: A Social and Behavioral Sciences Approach
Lee, H. K. H. (2008) · The American Statistician · doi:10.1198/000313008x370915 · Cited by 157
Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling (2nd Edition)
Stawski, R. S. (2013) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2013.797841 · Cited by 154
Empirical Strategies for Various Manifestations of Multilevel Data
Franzese, R. J. (2005) · Political Analysis · doi:10.1093/pan/mpi024 · Cited by 122
The Politicized Participant: Ideology and Political Action in 20 Democracies
van der Meer, T., van Deth, J. W., & Scheepers, P. L. H. (2009) · Comparative Political Studies · doi:10.1177/0010414009332136 · Cited by 120
Modeling Clustered Ordered Categorical Data: A Survey
Agresti, A., & Natarajan, R. (2001) · International Statistical Review · doi:10.1111/j.1751-5823.2001.tb00463.x · Cited by 116
Two-Step Hierarchical Estimation: Beyond Regression Analysis
Achen, C. H. (2005) · Political Analysis · doi:10.1093/pan/mpi033 · Cited by 104
Who Wants To Revise Privatization? The Complementarity of Market Skills and Institutions
DENISOVA, I., Eller, M., Frye, T., & Zhuravskaya, E. V. (2009) · American Political Science Review · doi:10.1017/s0003055409090248 · Cited by 81
Bayesian Multilevel Analysis and MCMC
Draper, D. (2007) · Handbook of Multilevel Analysis · doi:10.1007/978-0-387-73186-5_2 · Cited by 69
Sample size considerations in observational health care quality studies
Normand, S. T., & Zou, K. H. (2002) · Statistics in Medicine · doi:10.1002/sim.1020 · Cited by 40
Shields, L. E. (2007) · Nursing Children and Young People · doi:10.7748/paed.19.5.37.s32 · Cited by 39
Determination of Sample Size for Multilevel Model Design
Afshartous, D. R. (2011) · eScholarship (California Digital Library) · Cited by 16
Cited by
Explaining Fixed Effects: Random Effects Modeling of Time-Series Cross-Sectional and Panel Data
Bell, A., & Jones, K. (2015) · Political Science Research and Methods · doi:10.1017/psrm.2014.7 · Cited by 1,515
Fixed and random effects models: making an informed choice
Bell, A., Fairbrother, M., & Jones, K. (2019) · Quality & Quantity · doi:10.1007/s11135-018-0802-x · Cited by 1,085
Multilevel Modelling of Country Effects: A Cautionary Tale
Bryan, M. L., & Jenkins, S. P. (2016) · European Sociological Review · doi:10.1093/esr/jcv059 · Cited by 866
The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration
McNeish, D. M., & Stapleton, L. M. (2016) · Educational Psychology Review · doi:10.1007/s10648-014-9287-x · Cited by 715
On Using Bayesian Methods to Address Small Sample Problems
McNeish, D. (2016) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2016.1186549 · Cited by 570
Metadata
- Authors
- Stegmueller, D.
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2013-01-10
- Volume
- 57
- Issue
- 3
- Pages
- 748-761
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2157151897
- Cited by
- 818
- References
- 73
- FWCI
- 47.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.