The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Estimating Multilevel Logistic Regression Models When the Number of Clusters is Low: A Comparison of Different Statistical Software Procedures
Open accessAbstract
Multilevel logistic regression models are increasingly being used to analyze clustered data in medical, public health, epidemiological, and educational research. Procedures for estimating the parameters of such models are available in many statistical software packages. There is currently little evidence on the minimum number of clusters necessary to reliably fit multilevel regression models. We conducted a Monte Carlo study to compare the performance of different statistical software procedures for estimating multilevel logistic regression models when the number of clusters was low. We examined procedures available in BUGS, HLM, R, SAS, and Stata. We found that there were qualitative differences in the performance of different software procedures for estimating multilevel logistic models when the number of clusters was low. Among the likelihood-based procedures, estimation methods based on adaptive Gauss-Hermite approximations to the likelihood (glmer in R and xtlogit in Stata) or adaptive Gaussian quadrature (Proc NLMIXED in SAS) tended to have superior performance for estimating variance components when the number of clusters was small, compared to software procedures based on penalized quasi-likelihood. However, only Bayesian estimation with BUGS allowed for accurate estimation of variance components when there were fewer than 10 clusters. For all statistical software procedures, estimation of variance components tended to be poor when there were only five subjects per cluster, regardless of the number of clusters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.64”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 6,830
Approximate Inference in Generalized Linear Mixed Models
Breslow, N. E., & Clayton, D. G. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594284 · Cited by 4,662
Courgeau, D., & Goldstein, H. (1997) · Population · doi:10.2307/1534624 · Cited by 2,596
Approximate Inference in Generalized Linear Mixed Models
(1993) · doi:10.2307/2290687 · Cited by 1,800
Generalized linear mixed models a pseudo-likelihood approach
Wolfinger, R. D., & O'Connell, M. (1993) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659308811554 · Cited by 1,257
Estimation in Generalized Linear Models with Random Effects
Schall, R. (1991) · Biometrika · doi:10.1093/biomet/78.4.719 · Cited by 1,230
Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model
Pinheiro, J. C., Bates, D. M., & Pinheiro, J. C. (1995) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1995.10474663 · Cited by 1,171
A Language and Program for Complex Bayesian Modelling
Gilks, W., Thomas, A., & Spiegelhalter, D. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348941 · Cited by 760
Introducing Markov chain Monte Carlo
Gilks, W., Richardson, S., & Spiegelhalter, D. (1995) · Chapman & Hall/CRC eBooks · doi:10.1201/b14835-6 · Cited by 671
A comparison of Bayesian and likelihood-based methods for fitting multilevel models
Browne, W. J., & Draper, D. (2006) · Bayesian Analysis · doi:10.1214/06-ba117 · Cited by 603
An Assessment of Estimation Procedures for Multilevel Models with Binary Responses
Rodrı́guez, G., & Goldman, N. (1995) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2983404 · Cited by 472
Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model
Pinheiro, J. C., Bates, D. M., & Pinheiro, J. C. (1995) · Journal of Computational and Graphical Statistics · doi:10.2307/1390625 · Cited by 444
A note on Gauss—Hermite quadrature
LIU, Q., & Pierce, D. A. (1994) · Biometrika · doi:10.1093/biomet/81.3.624 · Cited by 350
Estimation in Generalized Linear Models with Random Effects
Schall, R. (1991) · Biometrika · doi:10.2307/2336923 · Cited by 203
Austin, P. C., Manca, A., Zwarenstein, M., Juurlink, D. N., & Stanbrook, M. B. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.002 · Cited by 163
Self-citationAn Introduction to Multilevel Regression Models
Austin, P. C., Goel, V., & van Walraven, C. (2001) · Canadian Journal of Public Health · doi:10.1007/bf03404950 · Cited by 146
Self-citationIntroduction to Multilevel Modeling
(2020) · Sage eBooks · doi:10.4135/9781544319315.n1 · Cited by 5
Research works printed but not linked: 3
- Bernardo JM, Berger JO, Dawid AP, Smith AFM), pp. 169-193. Oxford: Clarendon Press.
- Raudenbush SW (2008). Many small groups. In: Handbook of Multilevel Analysis (editors: de Leeuw J, Meijer E), pp. 207-236. New York, NY: Springer.
- The International Journal of Biostatistics, Vol. 6 [2010], Iss. 1, Art. 16 DOI: 10.2202/1557-4679.1195
Cited by
Sorted by citations
Multilevel Modelling of Country Effects: A Cautionary Tale
Bryan, M. L., & Jenkins, S. P. (2015) · European Sociological Review · doi:10.1093/esr/jcv059 · Cited by 874
The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration
McNeish, D., & Stapleton, L. M. (2014) · Educational Psychology Review · doi:10.1007/s10648-014-9287-x · Cited by 726
Burke, D. L., Ensor, J., & Riley, R. D. (2017) · Statistics in Medicine · doi:10.1002/sim.7141 · Cited by 538
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278
“Finally, because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.(64) For this reason, we presented several methods to overcome these 45 50 55…”
MethodModeling psychophysical data at the population-level: The generalized linear mixed model
Moscatelli, A., Mezzetti, M., & Lacquaniti, F. (2012) · Journal of Vision · doi:10.1167/12.11.26 · Cited by 224
Metadata
- Authors
- Austin, P. C.
- Affiliations
- Institute for Clinical Evaluative Sciences (CA)
- Corresponding author
- Austin, Peter C
- ORCID
- Austin, Peter C
- Venue
- The International Journal of Biostatistics
- Publisher
- Walter de Gruyter GmbH
- Published
- 2010-04-22
- Volume
- 6
- Issue
- 1
- Pages
- Article 16-Article 16
- ISSN
- 1557-4679 (ISSN-L)
- PMC
- PMC2949382
- OpenAlex
- W2093789600
- Cited by
- 193
- References
- 18
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 7 · 2023: 15 · 2022: 10 · 2021: 11 · 2020: 11 · 2019: 11 · 2018: 13 · 2017: 15 · 2016: 24 · 2015: 20 · 2014: 13 · 2013: 14 · 2012: 3
- FWCI
- 5.72
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The International Journal of Biostatistics (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Heart and Stroke Foundation of Canada
- Institute for Clinical Evaluative Sciences
- Ontario Ministry of Health and Long-Term Care
- Canadian Institutes of Health Research (86508, MOP86508)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- multilevel logistic regression, small number of clusters, variance component estimation, statistical software comparison, penalized quasi-likelihood, Bayesian estimation, likelihood estimation, clustered data, Monte Carlo simulation
- MeSH terms
- Bayes Theorem, Data Interpretation, Statistical, Humans, Monte Carlo Method, Software, Models, Statistical, Cluster Analysis, Linear Models, Logistic Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
