The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 1 of the 69 sources its claims cite were checked for support.
Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models
Open accessAbstract
In panel data models and other regressions with unobserved effects, fixed effects estimation is often paired with cluster-robust variance estimation (CRVE) to account for heteroscedasticity and un-modeled dependence among the errors. Although asymptotically consistent, CRVE can be biased downward when the number of clusters is small, leading to hypothesis tests with rejection rates that are too high. More accurate tests can be constructed using bias-reduced linearization (BRL), which corrects the CRVE based on a working model, in conjunction with a Satterthwaite approximation for t-tests. We propose a generalization of BRL that can be applied in models with arbitrary sets of fixed effects, where the original BRL method is undefined, and describe how to apply the method when the regression is estimated after absorbing the fixed effects. We also propose a small-sample test for multiple-parameter hypotheses, which generalizes the Satterthwaite approximation for t-tests. In simulations covering a wide range of scenarios, we find that the conventional cluster-robust Wald test can severely over-reject while the proposed small-sample test maintains Type I error close to nominal levels. The proposed methods are implemented in an R package called clubSandwich. This article has online supplementary materials.
Overview
CoChat Summary
TL;DR
The paper generalizes bias-reduced linearization (CR2) cluster-robust variance estimation to fixed effects models and proposes an approximate Hotelling’s T² (AHT) test for multi-parameter hypotheses. Simulations and two examples show the AHT test controls Type I error far better than standard CR1 Wald tests with \(m-1\) degrees of freedom, especially with few clusters or high-dimensional hypotheses.
Key findings
- The proposed CR2 adjustment remains well-defined in models with arbitrary fixed effects by using a Moore–Penrose inverse, and the AHT test extends Satterthwaite-type small-sample corrections to multi-parameter hypotheses.
- In simulations, the standard CR1 test with \(m-1\) df severely over-rejected: with 15 clusters, maximum Type I error ranged from 0.127 (\(q=1\)) to 0.687 (\(q=6\)); with 50 clusters, it still ranged from 0.071 (\(q=1\)) to 0.186 (\(q=6\)).
- The AHT test kept rejection rates near nominal: with \(m=15\), rates did not exceed 0.012 for \(\alpha=.01\), 0.055 for \(\alpha=.05\), and 0.106 for \(\alpha=.10\); with at least 30 clusters, AHT rates were between 0.019 and 0.054 across conditions.
- In the Achievement Awards example, the joint moderation test had standard \(p=.00010\) versus AHT \(p=.16057\); in the minimum legal drinking age example, the Hausman test had standard \(p=.06283\) versus AHT \(p=.11886\).
Methods
- Develops a generalized BRL/CR2 variance estimator for weighted least squares with arbitrary fixed effects, proves unbiasedness under a working model, and shows absorption equivalence for OLS with an identity working model; proposes the AHT test by approximating the cluster-robust Wald statistic with Hotelling’s T² and estimated degrees of freedom.
- Simulation study covered 1,944 unique conditions: randomized block, cluster-randomized, and difference-in-differences designs; balanced and unbalanced configurations; complete data or 15% missing; \(m=15,30,50\) clusters; main text states \(n=6,18,30\) units per cluster (Appendix Table A1 lists 12,18,30); 50,000 datasets per condition; tests of dimension \(q=1,2,3,6\) at \(\alpha=.05\).
- Empirical examples use the Achievement Awards randomized trial (35 female-student schools) and a state-by-year minimum legal drinking age panel for 1970–1983, comparing AHT tests to standard CR1 tests with \(m-1\) degrees of freedom.
Limitations
- Stated: AHT tests can under-reject when the number of clusters is very small and the design is unbalanced, severely so for multi-dimensional constraints (e.g., \(q=3\), \(m=15\), unbalanced difference-in-differences); CR2 is costly or infeasible when within-cluster sample sizes are large (e.g., \(n_i>10,000\)); extensions to multiway clustering, GEE, 2SLS, and GMM remain future work.
- Stated: Theorem 2 was revised to correct an error; the previous more general claim of WLS equivalence does not hold.
- Evident from the text: the main simulation description says \(n=6,18,30\) units per cluster, while Appendix Table A1 lists \(n=12,18,30\).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 4 Not valid · 45 Not checked · 83 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Evidence from a wide variety of contexts indicates that the asymptotic limiting distribution of robust Wald statistics can be a poor approximation when the number of clusters is small, even if corrections such as CR2 or CR3 are employed (Bell and McCaffrey 2002; Bertrand, Duflo, and Mullainathan 2004; Cameron, Gelbach, and Miller 2008).”
What the sources say
- Not matchedcites Bell and McCaffrey 2002
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Bertrand, Duflo, and Mullainathan 2004 sourceExperimental — accuracy not yet measured
Bertrand, Duflo and Mullainathan (2004) clearly support the core of the claim for their DD setting: the Introduction (p. 1) states that clustered-error solutions 'work well when the number of groups is large (e.g. 50 states) but fare poorly as the number of groups gets small'; §4.5 (p. 4.5) reports rejection rates of 9.5% with 20 states and 31.5% with only 6, §4.4 reports 15% with 6 states even under an arbitrary variance-covariance matrix, and footnote 31 (§4.6) notes 'at these small samples (as small as 12), the normal approximation to the t-distribution will clearly not work'. What our copy does NOT support is the qualifier 'even if corrections such as CR2 or CR3 are employed': no passage in our copy mentions CR2, CR3, or any small-sample degrees-of-freedom adjustment of that kind (the corrections examined are Moulton/White clustered SEs, AR(1) parametric, block bootstrap, aggregation, an arbitrary empirical variance-covariance matrix, and randomization inference). Likewise the framing 'a wide variety of contexts' overreaches this source, whose evidence is CPS-based DD minimum-wage simulations plus a survey of published DD papers, not a multi-context collection. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites Cameron, Gelbach, and Miller 2008 source
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The approximate degrees of freedom of the AHT test can be much smaller than the number of clusters, particularly when the covariates involved in the test involve high leverage or are unbalanced across clusters.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Accounting for fixed effects also makes hypothesis test results invariant to how the regression is calculated (whether by LSDV or after absorption), which we see as a useful feature for ensuring the replicability of one's analysis.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unbalanced designs lead to rejection rates that are usually below the nominal α -just the opposite of how the standard test is affected by unbalance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both of these designs, unbalance leads to sharply reduced degrees of freedom.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
81 more claims
Create a free account to see all 133 claims and to verify the 2 references still open.
- It may state two separate claims as one. 31
- Probably not a substantial claim. 20
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CR2 standard errors are designed for use with small-sample fixed effect models, and represent a conservative choice relative to conventional clustered standard errors.[46]”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To account for dependencies between effect sizes stemming from the same study and the small number of studies, we report estimates and standard errors based on cluster-robust covariance matrices (Pustejovsky & Tipton, 2018).”
From The Impact of COVID‐19‐Related School Closures on Student Achievement—A Meta‐Analysis (2022) · cites this paper as Pustejovsky & Tipton, 2018
The cited source says what the claim attributes to it.
In the citing paper
“we accounted for the hierarchical structure using robust inference”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Pustejovsky and Tipton, 2017
In the citing paper
“For many of the schemes only one to five studies are available, which does not allow for appropriate clustering of the effect sizes85,86.”
From Systematic review and meta-analysis of ex-post evaluations on the effectiveness of carbon pricing (2024) · cites this paper as [86]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.1 Design · p. 17
“with m = 15, 30, or 50 cluster and n = 6, 18, or 30 units per cluster”
§ Appendix C Details of simulation study · p. 40
“| Units per cluster ( n ) | 12, 18, 30 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1.3 Standard CRVE · p. 7
“where e i = Y i -X i ˆ β is the vector of residuals from cluster i”
§ 1.1 Econometric framework · p. 5
“Let U i = [ R i S i ] denote the set of predictors that are identified across multiple clusters, X i = [ R i S i T i ] denote the full set of predictors, α = ( β ′ , γ ′ , µ ′ ) ′ , and p = r + s + t .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 Achievement Awards demonstration · p. 25
“H 0 : β 21 = β 22 = β 23 , with q = 3”
§ 5.1 Achievement Awards demonstration · p. 26
“| Moderation - upper half (q = 2) | Standard | 3.186 | 34.00 | 0.05393 |”
Citations
Cites
(2001) · RePEc: Research Papers in Economics · Cited by 27,577
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550 · cited 2×
“A common approach to inference in these settings is to use a cluster-robust variance estimator (CRVE; White 1984; Liang and Zeger 1986; Arellano 1987).”
How Much Should We Trust Differences-in-Differences Estimates?
Bertrand, M., Duflo, E., & Mullainathan, S. (2004) · The Quarterly Journal of Economics · doi:10.1162/003355304772839588 · Cited by 10,641 · cited 3×
“The need to cluster the standard errors by state, even when including state fixed effects, was highlighted by Bertrand, Duflo, and Mullainathan (2004), who showed that to do otherwise can lead to inappropriately small standard errors and hypothesis tests with incorrect rejection rates.”
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467 · cited 3×
“In the past decade, use of CRVEs has become standard practice for micro-economic researchers, as evidenced by coverage in major textbooks and review articles (e.g., Angrist and Pischke 2009; Wooldridge 2010; Cameron and Miller 2015).”
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037 · cited 10×
“Following Cameron and Miller (2015), we refer to this estimator as CR0.”
InfluentialBootstrap-Based Improvements for Inference With Clustered Errors
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008) · Review of Economics and Statistics · doi:10.1162/rest.90.3.414 · Cited by 4,417 · cited 4×
“Cameron, Gelbach, and Miller (2008) investigated a range of bootstrapping procedures that provide improved Type I error control in small samples, finding that a cluster wild-bootstrap technique was particularly accurate in small samples.”
InfluentialAn Approximate Distribution of Estimates of Variance Components
Satterthwaite, F. E. (1946) · Biometrics Bulletin · doi:10.2307/3002019 · Cited by 4,373 · cited 1×
“ive t-test developed by Bell & McCarey (2002) involves using a t() references distribution, with degrees of freedom estimated by a Satterthwaite approximation. The Satterthwaite approximation (Satterthwaite 1946) entails using degrees of freedom that are a function of the the rst two moments of the sampling distri…”
Robust Inference With Multiway Clustering
Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2011) · Journal of Business & Economic Statistics · doi:10.1198/jbes.2010.07136 · Cited by 3,203 · cited 1×
“2 estimator and AHT testing procedure for weighted least squares estimation of linear regression models. Extensions to linear regression models with clustering in multiple, non-nested dimensions (cf. Cameron et al. 2011) appear to be possible, and their utility should be further investigated. McCarey & Bell (2006)…”
MacKinnon, J. G., & White, H. (1985) · Journal of Econometrics · doi:10.1016/0304-4076(85)90158-7 · Cited by 1,879 · cited 2×
“We refer to this correction as CR2 because it extends the HC2 variance estimator for regressions with uncorrelated errors, which is exactly unbiased when the errors are homoscedastic (MacKinnon and White 1985).”
Research works printed but not linked: 3
- Bell R. M., 2002, Survey Methodology, 28, 169
- Carter A. V., 2013, Technical Report, University of California, Santa Barbara, CA. Available at
- McCaffrey D. F., 2001, Proceedings of the Annual Meeting of the American Statistical Association
Cited by
Designing Difference in Difference Studies: Best Practices for Public Health Policy Research
Wing, C., Simon, K., & Bello-Gomez, R. A. (2018) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040617-013507 · Cited by 1,470
Cluster-robust inference: A guide to empirical practice
MacKinnon, J. G., Nielsen, M. Ø., & Webb, M. D. (2023) · Journal of Econometrics · doi:10.1016/j.jeconom.2022.04.001 · Cited by 407
“Young (2016) proposes a related method that uses CV1 instead of CV2.”
MethodInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
“, when effect sizes could be determined for both original and replication effects), we accounted for the hierarchical structure using robust inference (Bell and McCaffrey, 2002; Pustejovsky and Tipton, 2017) to account for the nesting structure as described in ‘Assessing candidate moderators of replication success met…”
MethodDigital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
clubSandwich: Cluster-Robust (Sandwich) Variance Estimators with Small-Sample Corrections
Pustejovsky, J. E. (2016) · CRAN: Contributed Packages · doi:10.32614/cran.package.clubsandwich · Cited by 216
Self-citation
Metadata
- Authors
- Pustejovsky, J. E., & Tipton, E.
- Affiliations
- University of Wisconsin–Madison (US)
- Columbia University (US)
- Venue
- Journal of Business & Economic Statistics
- Publisher
- Informa UK Limited
- Published
- 2017-06-12
- Volume
- 36
- Issue
- 4
- Pages
- 672-683
- ISSN
- 0735-0015 (ISSN-L)
- 1537-2707
- OpenAlex
- W2229226246
- Cited by
- 492
- References
- 46
- Citations by year
- 2026: 203 · 2025: 76 · 2024: 48 · 2023: 44 · 2022: 37 · 2021: 47 · 2020: 15 · 2019: 14 · 2018: 4 · 2017: 3
- FWCI
- 14.90
- Type
- journal article
- Language
- en
- Update notices
- correction (2023-02-22)
- Open access
- green
- arxiv.org
- Locations
- Journal of Business and Economic Statistics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Spatial and Panel Data Analysis
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Keywords
- cluster-robust variance estimation, fixed effects model, panel data, small sample inference, Satterthwaite approximation, Wald test, heteroscedasticity, over-rejection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models. https://app.cochat.ai/papers/10.1080/07350015.2016.1247004
- The paper
- Pustejovsky, J. E., & Tipton, E. (2018). Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models. Journal of Business & Economic Statistics. https://doi.org/10.1080/07350015.2016.1247004
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.