Measurement invariance conventions and reporting: The state of the art and future directions for psychological research

“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript indices and criteria should be used to judge “practical” differences in model fit between groups (e.g., Chen, 2007; Cheung & Rensvold, 2002).”

“Overall, the Δχ2 was reported for 62.6% of tests, the change in CFI was reported for 73.2% of tests, and the change in another alternative fit index (e.g., ΔRMSEA, ΔTLI, ΔSRMR) was reported for 56.1% of tests.”

“Perhaps tests of many groups rarely appear in the literature because fewer researchers study many groups, measurement invariance is not achieved (e.g., the file drawer problem; Rosenthal, 1979), or model complexity causes researchers to shy away from tests with many groups.”

“Vandenberg and Lance (2000) soon after outlined 8 steps for testing measurement invariance, with the first 5 steps consisting of the main tests of measurement invariance.”

“Steenkamp and Baumgartner (1998) suggested that ideally more than half of items on a factor should be invariant.”

“) Although a required component for full factorial invariance (Meredith, 1993), testing for residual invariance is not a prerequisite for testing mean differences because the residuals are not part of the latent factor, so invariance”

“Byrne, Shavelson, and Muthén (1989) described testing partial measurement invariance, but placed no restrictions on the number of parameters released other than nebulously suggesting that it “makes substantive sense to do so” (p. 465).”

“…A uthor M anuscript A uthor M anuscript A uthor M anuscript Criteria Used to Evaluate Model Fit Classically, measurement invariance was evaluated using a single criterion, significance of the change in χ2 for two nested models (Byrne et al., 1989; Marsh & Hocevar, 1985; Reise et al., 1993).”

“Using simulated data, Kenny, Kaniskan, and McCoach (2015) also found that the RMSEA is overly sensitive in models with few (e.g., 10) degrees of freedom.”

“, 2008), and some question the use of alternative fit indices entirely because of their lack of precision (Barrett, 2007; Bentler, 2007; Fan & Sivo, 2009). For sample sizes with adequate power, equal group sizes, and mixed invariance (i.e., some loadings are higher and some lower in the first group), Chen (2007) also…”

“Meade et al. (2008) recommended a ΔCFI < .002 based on the results of their Monte Carlo simulation, but Little (2013) suggested that the simulation parameters used by Meade and colleagues were too Dev Rev. Author manuscript; available in PMC 2017 September 01.”

“…is “uninformative with respect to the particular source of measurement inequivalence” (Vandenberg & Lance, 2000, p. 36), because contemporary guidelines now omit this first step (Milfont & Fischer, 2010; van de Schoot, Lugtig, & Hox, 2012), and because this test is rarely performed in practice.”

“To test measurement invariance, it is becoming more common to use a mean and covariance structure (MACS; Little, 1997; Ployhart & Oswald, 2004) framework, which includes item intercepts and factor means, rather than just item loadings and residual terms.”

“Classically, measurement invariance was evaluated using a single criterion, significance of the change in χ2 for two nested models (Byrne et al., 1989; Marsh & Hocevar, 1985; Reise et al., 1993).”

“To manage partial noninvariance, Chen (2008) suggested comparing the results of interest (e.g., mean differences across groups, regression coefficients between latent variables, etc.) using a partially invariant model (imposing constraints on invariant items only) to those using a fully invariant…”

“Could groups be analyzed separately and then aggregated as if across separate studies (e.g., Curran & Hussong, 2009)?”

“…CFI (Meade et al., 2008; Rutkowski & Svetina, 2014) or the use of other alternative fit indices (e.g., ΔRMSEA, ΔSRMR; Chen, 2007; Meade et al., 2008), and some question the use of alternative fit indices entirely because of their lack of precision (Barrett, 2007; Bentler, 2007; Fan & Sivo, 2009).”

“Schmitt and Kuljanin (2008) identified 75 articles about measurement invariance over a 8-year period from 2000-2007 (9.38 articles per year), and Johnson et al. (2009) reviewed a 3-year period from 2005-2007 and uncovered 153 articles on measurement invariance (51 articles per year).”

“…of a -.01 change in CFI for nested models is commonly used, but other researchers have suggested different criteria for CFI (Meade et al., 2008; Rutkowski & Svetina, 2014) or the use of other alternative fit indices (e.g., ΔRMSEA, ΔSRMR; Chen, 2007; Meade et al., 2008), and some question the…”

“…turn of the 21st century, methodologists increasingly directed attention to the significance of measurement invariance, especially within a structural equation modeling framework (Cheung & Rensvold, 1999, 2002; Little, 2000; Rensvold & Cheung, 1998; Steenkamp & Baumgartner, 1998; Vandenberg, 2002).”

“The most important future direction, to our minds, is grappling with the issue of sensitivity (Vandenberg, 2002), that is, the ability of measurement invariance tests to detect real, meaningful differences in constructs across groups or measurement occasions.”

“…may be tested in a single step (Muthén & Asparouhov, 2002), and models intended to compare SEM-CFA to IRT methods may also report a combined metric-and-scalar invariance step because discrimination and location parameters are conventionally tested together in IRT (e.g., Stark et al., 2006).”

“…of Measurement Invariance The conceptual importance of testing measurement invariance entered the literature more than 50 years ago (e.g., Meredith, 1964; Struening & Cohen, 1963), but statistical techniques for testing invariance have become more accessible to, and expected from, the…”

“…have shifted from a focus on absolute fit in terms of χ2 to a focus on alternative fit indices because χ2 is overly sensitive to small, unimportant deviations from a “perfect” model in large samples (Chen, 2007; Cheung & Rensvold, 2002; French & Finch, 2006; Meade, Johnson, & Braddy, 2008).”

“Readers interested in the IRT approach are referred to Tay, Meade, and Cao (2015) for an overview and tutorial, and Meade and Lautenschlager (2004) for a comparison of the SEM-CFA and IRT approaches.”

“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript In a Monte-Carlo simulation, Steinmetz (2013) demonstrated that metric noninvariance (unequal factor loadings) had a negligible effect on mean differences of a latent factor, but that scalar noninvariance (unequal…”

“In cases like these, researchers have begun to study methods, such as moderated nonlinear factor analysis (Bauer & Hussong, 2009; Molenaar, Dolan, Wicherts, & van der Maas, 2010) or score-based tests of measurement invariance (Wang, Merkle, & Zeleis, 2014), that allow for estimation of invariance…”

“A study may measure the same cognition or behavior (form) across groups or times, but that cognition or behavior can have a different meaning (function) for the different groups or at different times (Bornstein, 1995).”

“Similarly, Nye and Drasgow (2011) developed an effect size index, dMACS, to quantify the degree of measurement noninvariance of an item across groups, as well as estimates of the change in mean and variance of a factor as a result of noninvariance of an item.”

“Consequently, Cheung and colleagues (Cheung & Lau, 2012; Cheung & Rensvold, 1999; Rensvold & Cheung, 2001) described two procedures, the factor-ratio test and the stepwise portioning procedure, for testing pairs of items to identify those that are noninvariant.”

“To test measurement invariance, it is becoming more common to use a mean and covariance structure (MACS; Little, 1997; Ployhart & Oswald, 2004) framework, which includes item intercepts and factor means, rather than just item loadings and residual terms.”

“Guenole and Brown (2014) explored the effects of ignoring nonivariance across groups (e.”

“Readers interested in the IRT approach are referred to Tay, Meade, and Cao (2015) for an overview and tutorial, and Meade and Lautenschlager (2004) for a comparison of the SEM-CFA and IRT approaches.”

“…turn of the 21st century, methodologists increasingly directed attention to the significance of measurement invariance, especially within a structural equation modeling framework (Cheung & Rensvold, 1999, 2002; Little, 2000; Rensvold & Cheung, 1998; Steenkamp & Baumgartner, 1998; Vandenberg, 2002).”

“The χ2 statistic is sensitive to model size, and Herzog, Boomsma, and Reinecke (2007) recommended using a Swain correction to the χ2 (which corrects for sample size and model size) when large models are tested.”

“…turn of the 21st century, methodologists increasingly directed attention to the significance of measurement invariance, especially within a structural equation modeling framework (Cheung & Rensvold, 1999, 2002; Little, 2000; Rensvold & Cheung, 1998; Steenkamp & Baumgartner, 1998; Vandenberg, 2002).”

“Fan and Sivo (2009) questioned the use of change in alternative fit indices (e.g., ΔCFI, ΔRMSEA) because the performance of these tests was highly related to model size (e.g., tests were more sensitive in small models and much less sensitive as model size grew) in their Monte Carlo simulation.”

“A uthor M anuscript A uthor M anuscript A uthor M anuscript A uthor M anuscript is mixed regarding their ability to correctly identify invariant and noninvariant items (French & Finch, 2008; Jung & Yoon, 2016).”

“…Invariance The conceptual importance of testing measurement invariance entered the literature more than 50 years ago (e.g., Meredith, 1964; Struening & Cohen, 1963), but statistical techniques for testing invariance have become more accessible to, and expected from, the research community…”

“…before testing mean differences in a construct, researchers should test for invariance across child genders (Hong, Malik, & Lee, 2003), mothers and fathers (Wang et al., 2006), ethnic groups (Glanville & Wildhagen, 2007), and cultural groups (Senese, Bornstein, Haynes, Rossi, & Venuti, 2012).”

“To address this issue, Meuleman (2012) and Oberski (2014) suggested ways to compute the impact of a noninvariant parameter on the change in the estimated latent mean.”

“Schmitt and Kuljanin (2008) identified 75 articles about measurement invariance over a 8-year period from 2000-2007 (9.”

“Cheung and Rensvold (1998) describe a procedure for teasing apart the group difference attributable to item responses and unequal item-construct relations.”

“38 articles per year), and Johnson et al. (2009) reviewed a 3-year period from 2005-2007 and uncovered 153 articles on measurement invariance (51 articles per year).”

“Jung and Yoon (2016) compared backward, forward, and factor-ratio methods for identifying noninvariant parameters.”

“More research is needed on the performance of these tests across different conditions, and methods are currently only implemented using maximum likelihood estimation, but these procedures could be applied to other estimation methods in future research (Wang et al., 2014).”

“Yoon and Kim (2014) suggested that sequentially releasing constraints (backward method) based on the highest modification index has much smaller Type I (false positive) error rates than releasing all problematic constraints in a single pass.”

“To address this issue, Meuleman (2012) and Oberski (2014) suggested ways to compute the impact of a noninvariant parameter on the change in the estimated latent mean. Rather than relying on nested model comparisons or modification indices that are sensitive to sample size, Meuleman's (2012) and Oberski's (2014) proced…”

“…using the weighted least squares mean and variance adjusted estimator in Mplus (WLSMV; for estimating models with categorical variables; Muthén & Muthén, 2010) and multiple indicators multiple causes models (MIMIC; an alternative method of testing measurement invariance; Willse & Goodman, 2008).”

“We thus find support for weak measurement invariance and, in line with Putnick & Bornstein’s [59] recommendation, proceed with the analysis and interpret this as a minor limitation to our study. royalsocietypublishing.org/journal/rsos R.Soc.”

“Following Chen (2007); as cited in Putnick & Bornstein, 2016), the criteria we used for assessing invariance were ΔCFI < .01, ΔRMSEA < .015 and ΔSRMR < .030 (for weak invariance only) or ΔSRMR < .015 (for strong and strict invariance).”