The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Estimating Multilevel Logistic Regression Models When the Number of Clusters is Low: A Comparison of Different Statistical Software Procedures
Open accessAbstract
Multilevel logistic regression models are increasingly being used to analyze clustered data in medical, public health, epidemiological, and educational research. Procedures for estimating the parameters of such models are available in many statistical software packages. There is currently little evidence on the minimum number of clusters necessary to reliably fit multilevel regression models. We conducted a Monte Carlo study to compare the performance of different statistical software procedures for estimating multilevel logistic regression models when the number of clusters was low. We examined procedures available in BUGS, HLM, R, SAS, and Stata. We found that there were qualitative differences in the performance of different software procedures for estimating multilevel logistic models when the number of clusters was low. Among the likelihood-based procedures, estimation methods based on adaptive Gauss-Hermite approximations to the likelihood (glmer in R and xtlogit in Stata) or adaptive Gaussian quadrature (Proc NLMIXED in SAS) tended to have superior performance for estimating variance components when the number of clusters was small, compared to software procedures based on penalized quasi-likelihood. However, only Bayesian estimation with BUGS allowed for accurate estimation of variance components when there were fewer than 10 clusters. For all statistical software procedures, estimation of variance components tended to be poor when there were only five subjects per cluster, regardless of the number of clusters.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Finally, because validation studies of a certain prediction model are often sparse, traditional (frequentist) meta-analysis models may suffer from convergence issues and yield unreliable estimates of precision and between-study heterogeneity.64”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [64]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Hierarchical Linear Models: Applications and Data Analysis Methods
Davidian, M. (2003) · Journal of the American Statistical Association · doi:10.1198/jasa.2003.s288 · Cited by 13,820
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 6,830
Approximate Inference in Generalized Linear Mixed Models
Breslow, N. E., & Clayton, D. G. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594284 · Cited by 4,662
Courgeau, D., & Goldstein, H. (1997) · Population · doi:10.2307/1534624 · Cited by 2,596
Approximate Inference in Generalized Linear Mixed Models
(1993) · doi:10.2307/2290687 · Cited by 1,800
Generalized linear mixed models a pseudo-likelihood approach
Wolfinger, R. D., & O'Connell, M. (1993) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659308811554 · Cited by 1,257
Estimation in Generalized Linear Models with Random Effects
Schall, R. (1991) · Biometrika · doi:10.1093/biomet/78.4.719 · Cited by 1,230
Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model
Pinheiro, J. C., Bates, D. M., & Pinheiro, J. C. (1995) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1995.10474663 · Cited by 1,171
A Language and Program for Complex Bayesian Modelling
Gilks, W., Thomas, A., & Spiegelhalter, D. (1994) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348941 · Cited by 760
Research works printed but not linked: 3
- Bernardo JM, Berger JO, Dawid AP, Smith AFM), pp. 169-193. Oxford: Clarendon Press.
- Raudenbush SW (2008). Many small groups. In: Handbook of Multilevel Analysis (editors: de Leeuw J, Meijer E), pp. 207-236. New York, NY: Springer.
- The International Journal of Biostatistics, Vol. 6 [2010], Iss. 1, Art. 16 DOI: 10.2202/1557-4679.1195
Cited by
Sorted by citations
Items 101–191 of 191
Linear and Non-Linear Mixed Models in Longitudinal Studies and Complex Survey Data
Wang, K. S. (2016) · Journal of Biometrics & Biostatistics · doi:10.4172/2155-6180.1000290 · Cited by 11
Chatignoux, É., Remontet, L., Iwaz, J., Colonna, M., & Uhry, Z. (2018) · Biostatistics · doi:10.1093/biostatistics/kxy012 · Cited by 10
Smoking transmission in-home across three generations
Duarte, R., Escario, J., & Molina, J. (2015) · Journal of Substance Use · doi:10.3109/14659891.2015.1018970 · Cited by 9
van Ginkel, J. R., Juffer, F., Bakermans‐Kranenburg, M. J., & van IJzendoorn, M. H. (2016) · European Journal of Pediatrics · doi:10.1007/s00431-016-2697-7 · Cited by 9
Ahmed, K. R., Kolbe-Alexander, T. L., & Khan, A. (2023) · Public Health Nutrition · doi:10.1017/s1368980023002094 · Cited by 8
Frankot, M. A., Mueller, P. M., Young, M. E., & Haar, C. V. (2023) · Neuropsychopharmacology · doi:10.1038/s41386-023-01592-6 · Cited by 7
Bashkov, B. M., & DeMars, C. E. (2017) · Applied Psychological Measurement · doi:10.1177/0146621616688923 · Cited by 8
Rousselière, D., & Rousselière, S. (2016) · Public Understanding of Science · doi:10.1177/0963662515622394 · Cited by 8
Mohr, Z. T. (2013) · KU ScholarWorks (The University of Kansas) · Cited by 8
Investigating the skills of a preschool leader: A latent profile analysis
Held, L. L., Gilpin, A. T., & Xia, M. (2023) · Social Development · doi:10.1111/sode.12716 · Cited by 7
Gorelik, A. J., Paul, S. E., Miller, A. P., Baranger, D. A. A., Lin, S., Zhang, W., Elsayed, N. M., Modi, H., Addala, P., Bijsterbosch, J. D., et al. (2024) · Psychological Medicine · doi:10.1017/s0033291724002174 · Cited by 9
A Systematic Review and Comparative Study of R Packages for Ordinal Response Regression Models
Pujol-Rigol, S., Fernández, D., & Casals, M. (2025) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.70025 · Cited by 7
Keriel-Gascou, M., Buchet-Poyau, K., Duclos, A., Rabilloud, M., Figon, S., Dubois, J.-P., Brami, J., Vial, T., & Colin, C. (2013) · Implementation Science · doi:10.1186/1748-5908-8-69 · Cited by 8
Experimental Design and Data Analysis in Computer Simulation Studies in the Behavioral Sciences
Harwell, M. R., Kohli, N., & Peralta, Y. (2017) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1509494520 · Cited by 9
González, I. F., Carrillo, X., Martı́n, V., Hernández, J. M. D. L. T., Baz, J. A., Manchón, J. N., Masotti, M., Cequier, A. R., Cárdenas, M., & Manterola, F. A. (2015) · Revista Española de Cardiología (English Edition) · doi:10.1016/j.rec.2015.04.018 · Cited by 6
González, I. F., Carrillo, X., Martı́n, V., Hernández, J. M. D. L. T., Baz, J. A., Manchón, J. N., Masotti, M., Cequier, A. R., Cárdenas, M., & Manterola, F. A. (2015) · Revista Española de Cardiología · doi:10.1016/j.recesp.2015.04.018 · Cited by 6
Nunamaker, E. A., & Reynolds, P. S. (2022) · PLoS ONE · doi:10.1371/journal.pone.0274738 · Cited by 6
Using Hierarchical Logistic Regression to Study DIF and DIF Variance in Multilevel Data
Shear, B. R. (2018) · Journal of Educational Measurement · doi:10.1111/jedm.12190 · Cited by 6
Mohr, Z. T. (2017) · Public Administration Quarterly · doi:10.1177/073491491704100105 · Cited by 6
Regional Disparities in Family Environments, Preschool Quality and Early School Attainment in China
Rao, N., Yang, Y., Gong, J., Su, Y., Perlman, M., Sun, J., Zhang, L., & Zhou, J. W. (2025) · International Journal of Early Childhood · doi:10.1007/s13158-025-00418-4 · Cited by 5
An Illusory Motion in Stationary Stimuli Alters Their Perceived Duration
Contemori, G., Meneghini, G., & Battaglini, L. (2023) · Vision · doi:10.3390/vision7030061 · Cited by 5
Smoliński, P. R., & Brycz, H. (2023) · Personality and Individual Differences · doi:10.1016/j.paid.2023.112500 · Cited by 7
Job (In)Security and Workers’ Training Decisions: A Framing Approach
Kamphuis, P., & Glebbeek, A. C. (2020) · International Journal for Research in Vocational Education and Training · doi:10.13152/ijrvet.7.3.6 · Cited by 6
Gusha, M. N. C., Dalu, T., & McQuaid, C. D. (2021) · Hydrobiologia · doi:10.1007/s10750-021-04535-8 · Cited by 5
Determinants of Indefinite Contracts in Europe: The Role of Unemployment
Baccaro, L., Koçer, R. G., Galindo, J., & Pulignano, V. (2016) · Comparative Sociology · doi:10.1163/15691330-12341412 · Cited by 5
Monte-Carlo Simulation in Modeling for Hierarchical Generalized Linear Mixed Models
Irimata, K. M., & Wilson, J. R. (2017) · ICSA book series in statistics · doi:10.1007/978-981-10-3307-0_13 · Cited by 5
Osei, G. K., & Gorey, K. M. (2020) · Child and Adolescent Social Work Journal · doi:10.1007/s10560-020-00691-3 · Cited by 6
Bedaso, N. G., & Debusho, L. K. (2022) · BMC Infectious Diseases · doi:10.1186/s12879-021-06965-0 · Cited by 5
Rojanasarot, S., Carlson, A. M., Peter, W. L. S., Karaca‐Mandic, P., Wolfson, J. A., & Schommer, J. C. (2020) · Journal of Asthma · doi:10.1080/02770903.2020.1741611 · Cited by 5
Evaluating Possible Confounding by Prescriber in Comparative Effectiveness Research
Franklin, J. M., Schneeweiß, S., Huybrechts, K. F., & Glynn, R. J. (2015) · Epidemiology · doi:10.1097/ede.0000000000000241 · Cited by 5
Error accumulation when steering toward curves.
Goodridge, C. M., Billington, J., Markkula, G. M., & Wilkie, R. M. (2023) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/xhp0001101 · Cited by 9
Frankot, M. A., Young, M. E., & Haar, C. V. (2023) · Perspectives on Behavior Science · doi:10.1007/s40614-023-00388-9 · Cited by 4
Wang, Q., Li, C., Yu, S.-Y., Dong, H.-J., Yang, L., Liu, Y., He, P. F., Zhang, S., & Yu, Q. (2025) · Lara D. Veeken · doi:10.1093/rheumatology/keae706 · Cited by 4
Yesuf, K. A., Muluneh, A., & Asrese, E. T. (2020) · doi:10.21203/rs.3.rs-55326/v1 · Cited by 4
Yesuf, K. A., Kassa, A. M., & Asrese, E. T. (2020) · doi:10.21203/rs.3.rs-55326/v3 · Cited by 4
Gorelik, A. J., Paul, S. E., Karcher, N. R., Johnson, E. C., Nagella, I., Blaydon, L., Modi, H., Hansen, I. S., Colbert, S. M. C., Baranger, D. A. A., et al. (2023) · Behavior Genetics · doi:10.1007/s10519-023-10140-3 · Cited by 4
Zeleke, A. T. (2019) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20190803.11 · Cited by 3
Kirsch, K., Nagel, C., Iribagiza, C., Ecklu, J., Zawadi, G. A., Ntabaza, P. M., Barstow, C. K., Lund, A. J., Harper, J., Carlton, E. J., et al. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283019 · Cited by 4
The Impacts of Small Teacher-Level Sample Sizes in Cluster-Randomized Trials
Zhang, H., Shen, Z., & Leite, W. L. (2024) · The Journal of Experimental Education · doi:10.1080/00220973.2024.2376627 · Cited by 3
Engaged robots, disengaged workers: Automation and political alienation
González‐Rostani, V. (2024) · Economics and Politics · doi:10.1111/ecpo.12307 · Cited by 4
Chandra, A., & Verma, A. (2024) · Journal of the Air Transport Research Society · doi:10.1016/j.jatrs.2024.100053 · Cited by 3
Holm, J. R., & Lorenz, E. (2013) · VBN Forskningsportal (Aalborg Universitet) · Cited by 3
Evaluating the Quality of Point of Sale (POS) Software
Saeed, S., Nizamani, M. A., Shaikh, A., Memon, M. A., Abbasi, F. A., & Naqvi, S. M. R. (2019) · University of Sindh Journal of Information and Communication Technology · Cited by 3
Soziale Herkunft und die Umsetzung von Studienaspirationen
Claudia, C. (2022) · doi:10.3224/96665040 · Cited by 3
Smiley, W. F. (2015) · Scholar Commons (University of South Carolina) · Cited by 3
Hengelbrock, J., Konietschke, F., Herm, J., Audebert, H. J., & Aigner, A. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02497-2 · Cited by 2
Bayesian spatio-temporal modelling: a systematic review
Louzada, F., Diniz, C. A. R., Dawodu, O. A., & Egbon, O. A. (2025) · Research in Statistics · doi:10.1080/27684520.2025.2568891 · Cited by 2
Campbell, C., Bains, V., King, J., Doidge, J., van der Meulen, J., Walker, K., & Bernal, W. E. (2026) · Frontline Gastroenterology · doi:10.1136/flgastro-2026-103636 · Cited by 2
Promoting Graduation Outcomes for Racially Minoritized Law School Students
Bowman, N. D., Stroup, N. R., & Fenton-Miller, S. (2022) · Journal of Postsecondary Student Success · doi:10.33009/fsop_jpss130217 · Cited by 2
Is Major Minor? The Relationship Between Music Major Status and a Measure of Musical Sophistication
Morrison, S. J., Himes, A. D., Montemayor, M., & Springer, D. G. (2023) · Journal of Research in Music Education · doi:10.1177/00224294231185321 · Cited by 2
Hierarchical Logistic Regression Models
Wilson, J. R., Lorenz, K. A., & Selby, L. P. (2024) · ICSA book series in statistics · doi:10.1007/978-3-031-62427-8_10 · Cited by 1
Work organization, gender and workforce transition to non-employment in OECD countries
Калугина, Е., Greenan, N., & Niang, M. M. (2025) · Eurasian economic review : · doi:10.1007/s40821-025-00297-8 · Cited by 3
Federated Mixed Effects Logistic Regression Based on One‐Time Shared Summary Statistics
Limpoco, M. A. A., Faes, C., & Hens, N. (2025) · Biometrical Journal · doi:10.1002/bimj.70080 · Cited by 1
Kwak, J., & Wallace, M. (2023) · International Journal of Comparative Sociology · doi:10.1177/00207152231217749 · Cited by 1
European Welfare States: Does Decentralization Affect Poverty?
Coli, A., & Pacini, B. (2019) · Essentials · doi:10.1007/978-3-030-06022-0_21 · Cited by 1
Soziale Herkunft und die Umsetzung von Studienaspirationen
Finger, C. (2022) · doi:10.2307/j.ctv2b07vg5 · Cited by 1
Welfare State and Local Government: the Impact of Decentralization on Well-Being
Addis, P., Coli, A., & Pacini, B. (2016) · CINECA IRIS Institutial research information system (University of Pisa) · Cited by 1
Holm, J. R., & Lorenz, E. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2487849 · Cited by 1
Eldridge, S. M., & Kerry, S. (2012) · doi:10.1002/9781119966241.ch6 · Cited by 1
Yesuf, K. A., Kassa, A. M., & Asrese, E. T. (2020) · doi:10.21203/rs.3.rs-55326/v2 · Cited by 1
Error accumulation when steering toward curvesPreprint
Goodridge, C. M., Billington, J., Markkula, G. M., & Wilkie, R. M. (2022) · doi:10.31234/osf.io/2e7p4 · Cited by 1
Khanam, S. J., Alam, M. B., & Khan, M. N. (2023) · Research Square · doi:10.21203/rs.3.rs-3508309/v1 · Cited by 1
Meta-analysis of clinical prediction models
Debray, T. P. A. (2013) · Utrecht University Repository (Utrecht University) · doi:10.7490/f1000research.445.1
Duncan, S., Smith, M., Mavoa, S., Stewart, T. I., Ivory, V. C., McPhee, J. S., & Schofield, G. (2013)
Access to Trauma Centre Care in Canada - A National Comparison
Hill, A. D. (2013) · TSpace (University of Toronto)
Using Bayesian Multilevel Models to Control for Multiplicity among Means
Zweifel, M. J. (2018) · Lincoln (University of Nebraska)
Sharples, L. D., & Papachristofi, O. (2020) · Springer eBooks · doi:10.1007/978-3-319-52677-5_245-1
Iver, M. A. M., Wills, K. C., Sheldon, S. B., Clark, E., & Iver, D. J. M. (2021) · Grantee Submission
Riley, R. D., Tierney, J. F., & Stewart, L. A. (2021) · Wiley eBooks · doi:10.1002/9781119333784.refs2
Steyerberg, E. W. (2013) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.717996649.793474031
Sharples, L. D., & Papachristofi, O. (2022) · Springer eBooks · doi:10.1007/978-3-319-52636-2_245
Smoliński, P. R., & Brycz, H. W. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4482482
Multi-Level Analysis of Risk Factors for Intimate Partner Violence Against Males
Candia, D. A., Weke, P. G. O., & Manene, M. M. (2023) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20231204.12
To Charge or Not to Charge: Prosecutorial Characteristics and Responses to Auditor Reported Findings
Lau, M., & Flasher, R. (2025) · Journal of Forensic Accounting Research · doi:10.2308/jfar-2022-022
Kim, S. Y., Kim, G. S., Kim, J., Kim, H., Nho, J., & Kim, S. (2025) · Military Medicine · doi:10.1093/milmed/usaf339
Mohammed, M. O. M. (2025) · Research Square · doi:10.21203/rs.3.rs-7391348/v1
Tug, T., Mers, F., Schäkel, F., Höltig, D., Kreienbrock, L., & Ickstadt, K. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1611771
Hidalgo, A. S., Urreta, I., Bolíbar, I., Bassas, E., Bernaras, M. A., Sánchez, G. G., Cirera, D. S., Moraga, F. J. G., Gorostiza, I., Parra, R. D. F., et al. (2025) · Journal of Clinical Anesthesia · doi:10.1016/j.jclinane.2025.112033
Agyepong, L. A., Boateng, S. A., & Han, Z. (2025) · Journal of Risk Research · doi:10.1080/13669877.2025.2588576
Bedaso, N. G., & Debusho, L. K. (2022) · Figshare · doi:10.6084/m9.figshare.19418653
Corruption and ethnicity in Africa: the moderating role of institutional quality
Bonga‐Bonga, L., & Kirsten, F. (2025) · Development Studies Research · doi:10.1080/21665095.2025.2606691
Flexible Work Arrangements in Central and Eastern Europe: Effects on Organizational Outcomes
Berber, N., Stavrou, E., Slavić, А., & Gašić, D. (2025) · Journal of East European Management Studies · doi:10.31083/jeems44450
Authorship network bias in meta-analysis
Rieck, M., Mupepele, A., & Dormann, C. F. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2025.10063
Strunk, T., Mountain, J. A., Gordon, A., Deshpande, G., van den Boom, J., Paoli, A. G. D., Jacobs, S., Giannoutsos, N., Doherty, D. A., & Patole, S. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-108203
Zhang, Y., & Hubach, R. D. (2026) · Archives of Sexual Behavior · doi:10.1007/s10508-026-03412-6
Altawabeyeh, S., & Wynn, D. C. (2026) · Research in Engineering Design · doi:10.1007/s00163-026-00487-z
Pendse, V., Allam, N., Ngan, C. C., Lorette, E., Morin-Girard, I., & Andrysek, J. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001767
González, C. X. R., Nasri, B., Gagnon‐Dufresne, M., Saha, A., Pierce, L., Rahman, M., Mackfield, J. B., Zhang, Y., Barua, M., Rahman, M. M., et al. (2026) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0014802
Metadata
- Authors
- Austin, P. C.
- Affiliations
- Institute for Clinical Evaluative Sciences (CA)
- Corresponding author
- Austin, Peter C
- ORCID
- Austin, Peter C
- Venue
- The International Journal of Biostatistics
- Publisher
- Walter de Gruyter GmbH
- Published
- 2010-04-22
- Volume
- 6
- Issue
- 1
- Pages
- Article 16-Article 16
- ISSN
- 1557-4679 (ISSN-L)
- PMC
- PMC2949382
- OpenAlex
- W2093789600
- Cited by
- 193
- References
- 18
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 7 · 2023: 15 · 2022: 10 · 2021: 11 · 2020: 11 · 2019: 11 · 2018: 13 · 2017: 15 · 2016: 24 · 2015: 20 · 2014: 13 · 2013: 14 · 2012: 3
- FWCI
- 5.72
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- The International Journal of Biostatistics (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Heart and Stroke Foundation of Canada
- Institute for Clinical Evaluative Sciences
- Ontario Ministry of Health and Long-Term Care
- Canadian Institutes of Health Research (86508, MOP86508)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Keywords
- multilevel logistic regression, small number of clusters, variance component estimation, statistical software comparison, penalized quasi-likelihood, Bayesian estimation, likelihood estimation, clustered data, Monte Carlo simulation
- MeSH terms
- Bayes Theorem, Data Interpretation, Statistical, Humans, Monte Carlo Method, Software, Models, Statistical, Cluster Analysis, Linear Models, Logistic Models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
