Analysis of variance—why it is more important than ever

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The hierarchical regression analysis assumes exchangeable effects within batches, which is not necessarily best in every application; spatial models may be preferable for some data.

“The hierarchical regression analysis is based on the model of exchangeable effects within batches, as expressed in model (7), which is not necessarily the best analysis in any particular application.For example, Besag and Higdon (1999) recommend using spatial models (rather than exchangeable row and column effects) for data such as in the split-plot experiment described previously.”

• cites Besag and Higdon (1999) sourcePartly supported

  Besag and Higdon develop spatial models for agricultural field experiments and say that a model's invariance to arbitrary row and column effects is undesirable (section 2.3.2, page unavailable). This supports the general preference for spatial modelling, but the passages do not specifically recommend it over exchangeable row and column effects for the split-plot experiment. The statement about the citing authors’ hierarchical regression model is not attributed to Besag and Higdon. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

• AssumptionThe authors state this as an assumption: Each batch of regression coefficients is modeled as exchangeable normal effects with mean zero and its own variance.

May combine two claims.

For balanced designs, simple formulas exist, and effect estimates in a batch are averages of data adjusted to satisfy linear constraints.

“For balanced designs, however, simple formulas exist.We do not go through all the literature here [see, e.g., Cornfield and Tukey (1956), Green and Tukey (1960) and Plackett (1960)].A summary is given in Searle, Casella and McCulloch [(1992), Section 4.2].The basic idea is that, in a balanced design, the effect estimates β(m) j in a batch m are simply averages of data, adjusted to fit a set of”

• cites Cornfield and Tukey (1956) sourcePartly supported

  The Summary (p. 1) says the paper provides formulas for factorial analyses and describes them as not very complex, supporting the point that simple formulas exist. The Description of Situations Treated (pp. 11, 13) discusses averaging and normalization of contributions, but does not establish the claimed rule that estimates of effects in a batch are averages of observed data adjusted to satisfy linear constraints. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Green and Tukey (1960) sourceNot checked — no full text of the source
• cites Searle, Casella and McCulloch [(1992), Section 4.2] sourceNot checked — no full text of the source

May combine two claims.

For the split-plot data, three S-Plus aov attempts were made, and only the final one yielded correct comparisons.

“Figure 1 shows three attempts to fit the split-plot data with aov, only the last of which worked.”

• Also stated in: § 2.2.2. A simple hierarchical design. · plain

  “Three attempts at running the aov command in S-Plus.Only the last gave the correct comparisons.”

May combine two claims.

Ordinary nonhierarchical least-squares regression handles some ANOVA problems but struggles with hierarchical structures.

“As we show in Section 2 of this paper, least-squares regression solves some ANOVA problems but has trouble with hierarchical structures [see also Gelman (2000)].”

• cites Gelman (2000)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In the simple hierarchical design, adding 20 machine indicators as fixed effects creates collinearity; after dropping four, treatment standard errors still use within-machine variation and remain wrong.

“This suggests a model with 24 predictors: one constant, three treatment indicators, and 20 machine indicators.But these predictors are collinear, so we must eliminate four of the machine indicators.Unfortunately, the standard errors of the treatment effects in this model are estimated using the within-machine variation, which is still wrong.”

May combine two claims.

In the simple hierarchical design, two formulas yield identical estimated treatment effects but different standard errors and different ANOVA F-tests.

“Formulas (3) and ( 4) give the same estimated treatment effects but imply different standard errors and different ANOVA F-tests.”

May combine two claims.

In an example from Rubin and Gelman et al., the data are compatible with a hierarchical standard deviation of zero but also with values as high as 10 or 20.

“The data are consistent with a standard deviation of 0, but it could also be as high as 10 or 20.”

• cites Rubin (1981) sourceNot checked — no full text of the source
• cites Gelman, Carlin, Stern and Rubin [(1995), Chapter 5] sourceNot checked — no full text of the source

May combine two claims.

In the Web connect times data, company has a large effect on its own and in interaction with to, from, and in the three-way interaction.

“Company has a large effect on its own and, perhaps more interestingly, in interaction with to, from, and in the three-way interaction.”

May combine two claims.

In the 1988 election example, ethnicity*state*region interactions are surprisingly large, with differences between African-Americans and others varying dramatically by state.

“The ethnicity * state * region interactions are surprisingly large: the differences between African-Americans and others vary dramatically by state.”

May combine two claims.

In the 1988 election example, ethnicity is by far the most important demographic factor, and state also explains quite a bit of variation.

“Figure 6 displays the ANOVA table, which shows that ethnicity is by far the most important demographic factor, with state also explaining quite a bit of variation.”

May combine two claims.

In the Web connect times data, all main effects and almost all interactions are statistically significant.

“The ANOVA reveals that all the main effects and almost all the interactions are statistically significant.”

May combine two claims.

In the Web connect times data, from*hour has a relatively high estimated standard deviation but a relatively low mean square.

“For example, the effects of from * hour have a relatively high estimated standard deviation but a relatively low mean square (see, e.g., to * week).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Similar patterns were found for other quality dimensions in the survey.

“Similar patterns were identified for other quality dimensions measured in the survey.”

Classified as probably not a substantial claim.

In the authors' application areas, interest has been in how important effects of factor X are rather than whether factor X has an effect.

“In the application areas in which we have worked, interest has lain in questions of the form, “How important are the effects of factor X?,” rather than “Does factor X have an effect?””

Probably not a substantial claim: may not pass S3 (load-bearing).

The analysis problem becomes harder when the design is unbalanced.

“The problem becomes even more difficult if the design is unbalanced.”

Classified as probably not a substantial claim.

In complicated designs such as split-plot studies, constructing a suitable ANOVA is not always straightforward.

“Unfortunately, in complex problems (e.g., split-plot designs), it is not always easy to set up an appropriate ANOVA.”

May combine two claims.

Once the ANOVA rows are specified, hierarchical linear regression can compute ANOVA automatically and avoid the difficulties described earlier.

“Thus, the struggles of Section 2.2 are avoided, and (hierarchical) linear regression can indeed be used to compute ANOVA automatically, once the rows of the table (the sources of variation) have been specified.”

• AssumptionThe authors state this as an assumption: Each batch of regression coefficients is modeled as exchangeable normal effects with mean zero and its own variance.

May combine two claims.

ANOVA can be performed in multiple ways within classical linear regression, and some of those ways give incorrect answers.

“We include this not to disparage S-Plus in any way but just to point out that ANOVA can be done in many ways in the classical linear regression framework, and not all these ways give the correct answer.”

May combine two claims.

Batches with standard deviations near zero receive more shrinkage, which occurs for factors contributing little variation.

“There will be more shrinkage for batches of effects whose standard deviations σ m are near zero, which will occur for factors that contribute little variation to the data.”

• AssumptionThe authors state this as an assumption: Each batch of regression coefficients is modeled as exchangeable normal effects with mean zero and its own variance.

May combine two claims.

Design information is encoded in the linear regression design matrix, and this automatic encoding is especially useful for incomplete designs lacking a simple classical analysis.

“The second part of making the hierarchical ANOVA work is that the information from the design is encoded in the design matrix of the linear regression [as shown by Nelder (1965a, b) and implemented in the software Genstat].”

• Also stated in: § 3.3. Getting something for nothing? · plain

  “The automatic encoding is particularly useful in incomplete designs where there is no simple classical analysis.”

• cites Nelder (1965a, b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Classical ANOVA sometimes requires knowing the design to choose the correct analysis, while the hierarchical analysis does so automatically.

“In classical ANOVA, you (sometimes) need to know the design in order to select the correct analysis, as in the examples in Section 2.2.But the hierarchical analysis does it automatically.”

• AssumptionThe authors state this as an assumption: Each batch of regression coefficients is modeled as exchangeable normal effects with mean zero and its own variance.

May combine two claims.

The two variance parameters can share the same point estimate under classical unbiased estimation and under Bayesian inference with a noninformative prior, but the superpopulation variance has more uncertainty.

“The two variances can be given the same point estimate-in classical unbiased estimation E(s 2 m |σ 2 m ) = σ 2 m , and in Bayesian inference with a noninformative prior distribution (see Section 4.2) the conditional posterior mode of σ 2 m given all other parameters in the model is s 2 .The superpopulation variance has more uncertainty, however.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Finite-population inferences are more meaningful in some contexts, while superpopulation inferences are of interest in others.

“In some contexts (e.g., obtaining inference for the 50 U.S. states) the finite population seems more meaningful, whereas in others (e.g., subject-level effects in a psychological experiment) interest clearly lies in the superpopulation.”

Probably not a substantial claim: may not pass S2 (checkable).

Much of the fixed-effects and random-effects literature can be usefully reexpressed through finite-population and superpopulation inferences.

“As we discuss in Section 6, we believe that much of the literature on fixed and random effects can be fruitfully reexpressed in terms of finite-population and superpopulation inferences.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The superpopulation standard deviation describes uncertainty about predicting a new coefficient from a batch, while the finite-population standard deviation describes the existing coefficients in that batch.

“The superpopulation standard deviation characterizes the uncertainty for predicting a new coefficient from batch m, whereas the finite-population standard deviation describes the existing J m coefficients.”

May combine two claims.

In the hierarchical model, coefficient estimates are pulled toward zero, and the amount of shrinkage depends on the estimated variance components.

“From the hierarchical model the coefficient estimates will be pulled toward zero, with the amount of shrinkage determined by the estimated variance components.”

• AssumptionThe authors state this as an assumption: Each batch of regression coefficients is modeled as exchangeable normal effects with mean zero and its own variance.

May combine two claims.

When a batch has only two coefficients and much data, the coefficients and their finite-population variance are estimated accurately, but the superpopulation variance remains poorly estimated with one degree of freedom.

“To see the difference between the two variances, consider the extreme case in which J m = 2 [and so (df ) m = 1] and a large amount of data is available in both groups.Then the two parameters β (m) 1 and β (m) 2 will be estimated accurately and so will s 2 m = (β (m) 1β (m) 2 ) 2 /2.The superpopulation variance σ 2 m , on the other hand, is only being estimated by a measurement that is”

May combine two claims.

The hierarchical ANOVA still requires the modeler to specify the ANOVA rows and batching of linear parameters, but unlike classical analysis this specification is sufficient.

“First, as with the classical analyses, we require that the rows of the ANOVA be specified by the modeler.In the notation of ( 5) and ( 6), the user must specify the structuring or batching of the linear parameters β.In the classical analysis, however, this is not enough, as discussed in Section 2.2.”

May combine two claims.

Near-zero hierarchical variance components will be common in ANOVA settings, because large ANOVA tables are expected to contain near-zero higher-level interactions and because adding informative covariates reduces hierarchical variances.

“But we would argue that such situations will be common in ANOVA settings, for two reasons.First, when studying the many rows of a large ANOVA table, we expect (in fact, we hope) to see various nearzero variances at higher levels of interaction.”

• Also stated in: § 3.4. Classical and Bayesian interpretations. · plain

  “Our second reason for expecting to see near-zero variance components is that, as informative covariates are added to a linear FIG. 2. Illustration of the difficulties of point estimation for variance components.Pictured is the marginal posterior distribution for a hierarchical standard deviation parameter from Rubin (1981) and Gelman, Carlin, Stern and Rubin [(1995), Chapter 5].The simplest point”

• cites Gelman (1996)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Fitting a multilevel model automatically gives each factor or interaction row a variance component, so sparse categories are not a problem.

“This is not a problem, however, if a multilevel model is fit, as is done automatically in our ANOVA procedure: each factor or set of interactions in the model, corresponding to a row in the ANOVA table, is automatically given a variance component.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Including contrasts requires specifying a distribution over the contrast variable to define a superpopulation for predictive purposes.

“In either case, including contrasts adds another twist in that defining a superpopulation for predictive purposes now requires specifying a distribution over the contrast variable (e.g., in the Latin square example, if the rows are labeled as -2, -1, 0, 1, 2, then a reasonable superpopulation might be a uniform distribution on the range [-2.5, 2.5]).”

May combine two claims.

For effects that are unique or in pairs (df=1), the posterior density for σ_m is improper with infinite mass as σ_j→∞, so coefficients in these batches are essentially estimated by maximum likelihood.

“However, for effects that are unique or in pairs [i.e., batches for which (df ) m = 1], the posterior density for the corresponding σ m is improper, with infinite mass in the limit σ j → ∞ [Gelman, Carlin, Stern and Rubin (1995), Exercise 5.8], and so the coefficients β (m) j in these batches are essentially being estimated via maximum likelihood.”

• cites Gelman, Carlin, Stern and Rubin (1995) sourceNot checked — no full text of the source

May combine two claims.

The simulation approach has a parametric bootstrap or Bayesian flavor and is motivated by approximate equivalence between repeated-sampling and Bayesian inferences.

“This simulation has a parametric bootstrap or Bayesian flavor and is motivated by the approximate equivalence between repeated-sampling and Bayesian inferences [see, e.g., DeGroot (1970) and Efron and Tibshirani (1993)].”

• cites DeGroot (1970)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Efron and Tibshirani (1993) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

Given σ, the parameters β(m)j have a multivariate normal distribution, which is a conditional posterior distribution in Bayesian terms or a predictive distribution in classical terms.

“Given σ , the parameters β (m) j have a multivariate normal distribution (in Bayesian terms, a conditional posterior distribution; in classical terms, a predictive distribution).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Hypothesis testing is generally more difficult than estimation because many different hypotheses can be considered.

“Hypothesis testing is in general a more difficult problem than estimation because many different possible hypotheses can be considered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The alternative ANOVA display can be formatted to occupy no more space than the classical ANOVA table.

“In addition, using careful formatting, we can display this in no more space than is required by the classical ANOVA table.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the election survey, state-level predictors such as previous Presidential election results can be added.

“In the election survey example, one can add state-level predictors such as previous Presidential election results.”

Classified as probably not a substantial claim.

A crucial difficulty in classical ANOVA and linear modeling is identifying the correct variance components for standard errors and hypothesis tests.

“A crucial difficulty in classical ANOVA and, more generally, in classical linear modeling, is identifying the correct variance components to use in computing standard errors and testing hypotheses.”

May combine two claims.

In the ANOVA formulation, regression error terms are the highest-level interactions, so the distributions for β include both likelihood and prior.

“In our ANOVA formulation (5) or (6), the regression error terms are just the highest-level interactions, β (M) j , and so the distributions (7) include the likelihood as well as the prior distribution.”

Probably not a substantial claim: may not pass S2 (checkable).

ANOVA ideas are extremely powerful in modeling complex data.

“Conversely, the ideas of ANOVA are extremely powerful in modeling complex data of the sort that we increasingly handle in statistics-hence the title of this paper.”

Classified as probably not a substantial claim. May combine two claims.

Analysis of variance is fundamentally about multilevel modeling, with each row corresponding to a batch of parameters and inference about their standard deviation.

“Analysis of variance is fundamentally about multilevel modeling: each row in the ANOVA table corresponds to a different batch of parameters, along with inference about the standard deviation of the parameters in this batch.”

May combine two claims.

Conditional on simulated σ, finite-population standard deviations s_m can be estimated, and their intervals are narrower than those for σ_m, especially when variance components have few degrees of freedom.

“Conditional on the simulation for σ , we can now estimate the finite-population standard deviations s m .As discussed in Section 3.5, the data provide additional information about these, and so our intervals for s m will be narrower than for σ m , especially for variance components with few degrees of freedom.”

Classified as probably not a substantial claim.

The alternative ANOVA table presents estimates and uncertainties for s_m for each source of variation.

“Our alternative ANOVA table presents, for each source of variation m, the estimates and uncertainties for s m , the standard deviation of the coefficients corresponding to that row of the table.”

May combine two claims.

On the logarithmic scale, standard deviation parameters can be interpreted directly; s_m=0.20 corresponds to a coefficient of variation of about 0.2 on the original scale.

“Since the data are on the logarithmic scale, the standard deviation parameters can be interpreted directly.For example, s m = 0.20 corresponds to a coefficient of variation of exp(0.2) -1 ≈ 0.2 on the original scale, and so the unlogged coefficients exp(β (m) j ) in this batch correspond to multiplicative increases or decreases in the range of 20%.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The model can be expanded by generalizing effect distributions beyond normality and by modeling variance parameters hierarchically.

“Most simply, the distributions for the effects in each batch can be generalized beyond normality (e.g., using t or mixture distributions), and the variance parameters can themselves be modeled hierarchically, as discussed immediately above.”

May combine two claims.

Classical hierarchical ANOVA computations reduce to estimating expected mean squares from estimated variance components, and for unbalanced designs this can be more complicated than the Bayesian computation.

“Thus, the classical hierarchical ANOVA computations reduce to estimating the expected mean squares EMS m (and thus EV m ) in terms of the estimated variance components σ m .For nonbalanced designs, this can be complicated compared to the Bayesian computation as described in Section 4.2.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In some simple balanced designs, hypotheses can be tested independently; the split-plot Latin square allows independent testing of row, column, and treatment effects at between- and within-plot levels.

“In some relatively simple balanced designs, the hypotheses can be tested independently; for example, the split-plot Latin square allows independent testing of row, column and treatment effects at the between-and within-plot levels.”

Probably not a substantial claim: may not pass S4 (based).

Assigning probability distributions for all variance components automatically gives correct comparisons and standard errors.

“However, as we discuss in this paper, assigning probability distributions for all variance components automatically gives the correct comparisons and standard errors.”

May combine two claims.

Reparameterizing into γ, α, and τ reduces to the original model and allows more efficient updating.

“A more efficient updating reparameterizes into vectors γ , α and τ , which are defined as follows:”

• Also stated in: § 4.2. Bayesian inference. · plain

  “The auxiliary parameters α are given a uniform prior distribution, and then this reduces to the original model [see Boscardin (1996), Meng and van Dyk (1997), Liu, Rubin and Wu (1998), Liu and Wu (1999) and Gelman (2004)].”

• cites Boscardin (1996)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Meng and van Dyk (1997) sourceNot checked — no full text of the source
• cites Liu, Rubin and Wu (1998)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For the Web data, finite-population standard deviations are appropriate for comparing the two companies and 25 hours, but superpopulation variances would be relevant for generalizing to the larger populations of to sites, from sites, and weeks.

“This makes sense when comparing the two companies and 25 hours, but the “to” sites, the “from” sites and the weeks are sampled from a larger population, and for these generalizations, the superpopulation variances would be relevant.”

Classified as probably not a substantial claim.

It is possible for a factor to be fixed under some definitions and random under others.

“In summary, it is easily possible for a factor to be “fixed” according to some of the definitions above and “random” for others.”

Probably not a substantial claim: may not pass S3 (load-bearing).

From a classical perspective, the two examples demonstrate the effectiveness of automatically setting up hierarchical models with random effects for each row in the ANOVA table.

“From a classical perspective of inference for variance components, these cases can be considered as examples of the effectiveness of automatically setting up hierarchical models with random effects for each row in the ANOVA table.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Latin square, ordered rows, columns, and treatments could be modeled with regression coefficients for linear trends.

“For example, the five rows, columns and possibly treatments in the Latin square are ordered, and systematic patterns there could be modeled, at the very least, using regression coefficients for linear trends.”

May combine two claims.

For large J_m, σ_m is essentially estimated from data; for small J_m, a flat prior allows σ to take large values, minimizing shrinkage in effect estimates.

“For values of m in which J m is large (i.e., rows of the ANOVA table corresponding to many linear predictors), σ m is essentially estimated from data.When J m is small, the flat prior distribution implies that σ is allowed the possibility of taking on large values, which minimizes the amount of shrinkage in the effect estimates.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Classical shrinkage estimation dominates least squares when estimating three or more parameters in a normal model.

“This relates to the classical result that shrinkage estimation dominates least squares when estimating three or more parameters in a normal model [James and Stein (1961)].”

• cites James and Stein (1961)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Different and incompatible definitions of fixed and random effects are used in different contexts.

“It turns out that different-in fact, incompatible-definitions are used in different contexts.[See also Kreft and de Leeuw (1998), Section 1.3.3,for a discussion of the multiplicity of definitions of fixed and random effects and coefficients, and Robinson (1998) for a historical overview.]”

• cites Kreft and de Leeuw (1998) sourceNot checked — no full text of the source
• cites Robinson (1998)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In the alternative ANOVA table, inferences for all variance components are simultaneous, unlike the classical approach where each variance component is tested assuming all others except error are zero.

“In our ANOVA table, the inferences for all the variance components are simultaneous, in contrast to the classical approach in which each variance component is tested under the model that all others, except for the error term, are zero.”

May combine two claims.

The mean square is variance explained per parameter and is not directly comparable to s^2_m and σ^2_m.

“The mean square is the variance explained per parameter, which is not directly comparable to the parameters s 2 m and σ 2 m , which represent underlying variance components.”

Classified as probably not a substantial claim.

In the paper's terminology, varying effects occur in batches regardless of whether effects are interesting in themselves or are a sample from a larger set.

“In this terminology (which we would apply at any level of the hierarchy in a multilevel model), varying effects occur in batches, whether or not the effects are interesting in themselves (definition 2), and whether or not they are a sample from a larger set (definition 3).”

Probably not a substantial claim: may not pass S3 (load-bearing).

After subtracting batch-level predictors, additive effects for factor levels in each batch could be modeled as exchangeable.

“After subtracting batch-level regression predictors, the additive effects for the factor levels in each batch could be modeled as exchangeable.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Definitions 4 and 5 do not arise for the paper because it estimates all batches hierarchically with variance components estimated from data.

“Definitions 4 and 5 do not arise for us since we estimate all batches of effects hierarchically, with the variance components σ m estimated from data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This corresponds to analysis of covariance or contrast analysis in classical ANOVA.

“This corresponds to analysis of covariance or contrast analysis in classical ANOVA.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's model for linear parameters corresponds to default ANOVA inferences based on variances and exchangeable coefficients within each batch.

“Our model (7) for the linear parameters corresponds to the default inferences in ANOVA, based on computations of variances and exchangeable coefficients within each batch.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the ANOVA context, such a model is potentially more difficult to set up because it should reflect the structure of variance components.

“In the ANOVA context such a model is potentially more difficult to set up since it should ideally reflect the structure of the variance components (e.g., if two sets of main effects are large, then one might expect their interaction to be potentially large).”

May combine two claims.

For balanced designs, variance components can be estimated noniteratively from the bottom of the table upward, or by solving a linear system for the whole vector at once.

“For balanced designs, then, variance components can be estimated by starting at the bottom of the table (with the highest-level interaction, or residuals) and then working upwards, at each step using the appropriate variance components from lower in the table in formulas ( 10) and ( 11).In this way the variance components σ 2 m can be estimated noniteratively.Alternatively, we can compute the”

May combine two claims.

Uncertainties can be much lower for finite-population variances than for superpopulation variances, and finite-population variances connect to classical ANOVA where useful inferences are possible for small batches.

“As discussed in Section 3.5, uncertainties can be much lower for finite-population variances s 2 m than for superpopulation variances σ 2 m , and it is through finitepopulation variances that we connect to classical ANOVA, in which it is possible to draw useful inferences for even small batches (as in our split-plot Latin square example).”

May combine two claims.

The inferential procedure described by Gelman and Little (1997) and Bafumi et al. (2002) works well and outperforms standard survey estimates for state-level outcomes.

“As described by Gelman and Little (1997) and Bafumi, Gelman and Park (2002), this inferential procedure works well and outperforms standard survey estimates when estimating state-level outcomes.”

• cites Gelman and Little (1997)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Bafumi, Gelman and Park (2002)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

With only five treatment levels, nonzero superpopulation variances are inherently difficult to estimate.

“with only five treatment levels, nonzero superpopulation variances are inherently difficult to estimate”

Classified as probably not a substantial claim.

ANOVA is intended to add perspective to, not replace, inference for quantities of substantive interest.

“ANOVA may be “more important than ever” but it is intended to add perspective to, not to take the place of, inference for quantities of substantive interest.”

May combine two claims.

In the special neoclassical model, individual treatment effects are not parameters but can be estimated by prediction.

“Individual treatment effects do not occur as parameters in this model, but they may be estimated by prediction”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The authors' procedure gives an appropriate answer in a wide range of classical problems.

“Our procedure gives an appropriate answer in a wide range of classical problems”

Classified as probably not a substantial claim.

The geographic variation finding implies that quality improvement interventions may need to target providers in an area rather than lower-performing health plans.

“This finding has implications for quality improvement, suggesting that interventions to improve quality of care might have to be directed to health care providers in an area rather than trying to identify and improve lower-performing health plans.”

• AssumptionRests on an assumption: Health care is largely provided by doctors and hospitals that are organizationally independent of health plans and may contract with multiple plans.

Classified as probably not a substantial claim.

The multilevel analysis approach used by Hox and Hoijtink contradicts the notion that hierarchical modeling automatically leads to a correct model.

“It certainly contradicts the notion implied in Sections 3.2 and 3.3 that using hierarchical modeling, so to speak, automagically leads to a correct model.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Statisticians are generally unaware of the many conflicting definitions of "fixed" and "random" effects.

“We suspect that statisticians are generally unaware of the many conflicting definitions of the terms “fixed” and “random””

Probably not a substantial claim: may not pass S4 (based).

The simple decomposition of Gelman's equation (1) applies only when unbiased estimators with independent errors exist, which is unlikely for complex models with unbalanced datasets.

“the simple decomposition of Gelman's equation (1) only applies when there are unbiased estimators with independent errors, which is not likely to be the case for complex models with unbalanced datasets.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Posterior predictive checking is the best approach for hypothesis testing because it allows testing any subset of parameters while accounting for uncertainty in other parameters.

“we agree with Zaslavsky that posterior predictive checking is the best approach, since it allows a hypothesis about any subset of parameters to be tested while accounting for uncertainty in the estimation of the other parameters in the model.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Researchers in psychology use ANOVA in a multilevel modeling way but are often ill-served by the classical framework of mean squares and F-tests.

“This is the way that researchers in psychology use ANOVA, but they are often ill-served by the classical framework of mean squares and F-tests.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A block factor cannot be transformed into a treatment factor because blocks are unlabelled.

“A block factor cannot be transformed into a treatment factor because the blocks are unlabelled.”

Classified as probably not a substantial claim.

The transition from ANOVA to variance components modeling shifts attention from decomposition of sample variance to decomposition of population variance.

“The transition from ANOVA to variance components modeling shifts attention from decomposition of the variance of the sample as in classical ANOVA to decomposition of the variance of the population.”

Probably not a substantial claim: may not pass S4 (based).

Scale-invariant improper priors typically yield degenerate posteriors in variance components models.

“Scale-invariant improper priors typically yield degenerate posteriors in variance components models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Each exchangeable variance-components model is a trivial representation in the space of symmetric matrices that is closed under deletion of levels.

“each exchangeable variance-components model is a trivial representation in the space of symmetric matrices that is closed under deletion of levels.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The term "fixed effect" has at least five other existing interpretations in the statistical literature.

“an already overloaded expression that has at least five other existing interpretations in the statistical literature!”

Probably not a substantial claim: may not pass S4 (based).

The parameters in a model are not just a long undifferentiated vector but can be usefully grouped into batches, which are already specified in the classical ANOVA table.

“our basic idea is to recognize that the parameters in a model are not simply a long undifferentiated vector but can be usefully grouped into batches, which in fact are already specified in the classical ANOVA table.”

May combine two claims.

ANOVA ideas help include finite-population and superpopulation inferences in a single fitted model, unifying fixed and random effects.

“The ideas of the analysis of variance also help us to include finite-population and superpopulation inferences in a single fitted model, hence unifying fixed and random effects.”

Probably not a substantial claim: may not pass S2 (checkable).

ANOVA is more important than ever because it represents the key idea of grouping predictor variables and coefficients into batches.

“ANOVA is more important than ever because it represents a key idea in statistical modeling of complex data structures-the grouping of predictor variables and their coefficients into batches.”

Probably not a substantial claim: may not pass S4 (based).

ANOVA decomposition requires an inner product on the observation space.

“ANOVA decomposition requires one further ingredient in the form of an inner product on the observation space.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Inference can be made for both finite-population and superpopulation variances, and this distinction corresponds to fixed versus random effects.

“Gelman correctly notes that inference can be made for both finite-population and superpopulation variances, and that the distinction between these two targets of inference corresponds to the distinction between fixed and random effects.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Variance components can be interpreted to nonstatisticians in a scientifically meaningful way.

“These examples illustrate that despite their relative unfamiliarity in many fields, variance components can be interpreted to nonstatisticians in a scientifically meaningful way.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

If one is already fitting a statistical model, its parameters can probably be grouped into batches, and comparing the magnitude of variation across batches is probably interesting.

“if you are already fitting a statistical model, its parameters can probably be grouped into batches, and it is probably interesting to compare the magnitude of the variation of the parameters in each batch.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Posterior predictive checking can be applied to the multilevel model as a whole to test assumptions such as additivity, linearity and normality.

“Posterior predictive checking can also be applied to the multilevel model as a whole to test assumptions such as additivity, linearity and normality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Nonadditive structures of interactions are an interesting class of generalizations of classical ANOVA.

“One particularly interesting class of generalizations of classical ANOVA involves the nonadditive structures of interactions.”

Classified as probably not a substantial claim.

Prior studies have found substantial geographic variation in health care providers' practice patterns.

“other studies have shown substantial geographical variation in their practice patterns.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In any specific application, an applied researcher will typically be interested in particular treatment effects or comparisons rather than variance components.

“in any specific application an applied researcher will typically be interested in particular treatment effects, or comparisons of treatment effects, rather than in variance components.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The subspace determined by a factor is a ring closed under functional multiplication with 1 as identity element.

“The subspace A ⊂ R n determined by a factor is of a very special type: it is also a ring, closed under functional multiplication, with 1 as identity element.”

Classified as probably not a substantial claim.

The authors find within-batch standard deviation summaries more relevant than the usual ANOVA table of sums of squares, mean squares and F-tests.

“we find the summary in terms of within-batch standard deviations to be more relevant than the usual ANOVA table of sums of squares, mean squares and F-tests.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The term "random effect" has no clear or formal definition.

“The term “random effect” has no clear (let alone “formal”) definition”

Classified as probably not a substantial claim. May combine two claims.

Treating treatment effects as fixed versus random requires knowledge of the actual design and a decision on how to view the treatments.

“Treating the treatment effects as fixed versus random requires knowledge about the actual design of the study, and a decision on how we should view these treatments.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Batches of parameters should not necessarily be modeled exchangeably.

“such batches should not necessarily be modeled exchangeably”

May combine two claims.

The boundary problem is not an issue for posterior predictive testing because the reference distribution is determined by simulation and no prior is required for the tested variance component.

“The boundary problem is not an issue with this approach since the reference distribution is determined by simulation rather than by asymptotics, and indeed no prior distribution is required for the variance component being tested.”

Probably not a substantial claim: may not pass S4 (based).

A joint prior distribution on variance components avoids the discontinuity associated with near-zero estimated variance components.

“A joint prior distribution on the variance components avoids the discontinuity associated with near-zero estimated variance components.”

May combine two claims.

Gelman's prior distribution is uninformative and lacks competing models or different sources of prior information evaluated via posterior probabilities.

“His prior distribution is uninformative, and there are neither competing models nor different sources of prior information that are evaluated using posterior probabilities.”

May combine two claims.

In the Medicare survey data, effects of measured characteristics of individual members or plans are fairly small, and individual characteristics are of little interest because the survey's primary objective is to evaluate health care systems.

“The effects of measured characteristics of individual members or plans are fairly small, and the effects of an individual's characteristics are of little interest because the primary objective of the survey is to evaluate health care systems, not the predictors of an individual's reported experiences.”

• cites Zaslavsky, Zaborski and Cleary (2004) sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

In a Bayesian framework, estimation of variance components is relatively automatic, allowing attention to focus on defining sensible models rather than constructing easily analyzable designs.

“As Gelman notes, in a Bayesian framework the estimation of variance components is relatively automatic; attention can be focused on defining sensible models rather than on constructing designs that can be analyzed easily.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A Taguchi-type model where variance depends on factor levels is a representation but not a trivial one.

“By contrast, a Taguchi-type model in which the variance depends on one or more factor levels is a representation, but not a trivial representation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

If one is already fitting a complicated model, one's inferences can be better understood using the structure of that model.

“if you are already fitting a complicated model, your inferences can be better understood using the structure of that model.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MLwiN includes a fully Bayesian inference option, though maximum likelihood is not attractive given the small sample size.

“Given the small sample size, maximum likelihood estimation is not attractive, but MLwiN includes a fully Bayesian inference option.”

• cites Goldstein et al. (1998)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

All models discussed by Gelman for the example data can be analyzed using MLwiN.

“all models discussed by Gelman for these data can be analyzed using the software MLwiN [Goldstein et al. (1998)].”

• cites Goldstein et al. (1998)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.