A cautionary note on testing latent variable models
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The article tackles the practice of testing latent variable models. The analysis covered recently published studies from 11 psychology journals varying in orientation and impact. Seventy-five studies that matched the criterion of applying some of the latent modeling techniques were reviewed. Results indicate the presence of a general tendency to ignore the model test (χ(2)) followed by the acceptance of approximate fit hypothesis without detailed model examination yielding relevant empirical evidence. Due to reduced sensitivity of such a procedure to confront theory with data, there is an almost invariable tendency to accept the theoretical model. This absence of model test consequences, manifested in frequently unsubstantiated neglect of evidence speaking against the model, thus implies the perilous question of whether such empirical testing of latent structures (the way it is widely applied) makes sense at all.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
Power analysis and determination of sample size for covariance structure modeling.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.130 · Cited by 10,418
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Therefore, satisfactory values of AFI indicating close fit (e.g., CFI ≥ 0.95) do not ensure that the model failed the χ2 test because of a trivial misspecification (Olsson et al., 2000; Tomarken and Waller, 2003; McIntosh, 2012; Hayduk, 2014a).”
InfluentialMarsh, H. W., Hau, K.-T., & Wen, Z. (2004) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1103_2 · Cited by 6,691 · cited 4×
“…2005); (2) with several frequently used AFI, the probability of correctly rejecting misspecified models systematically decreases with growing N (Marsh et al., 2004; Sharma et al., 2005) which is, paradoxically, the very opposite as with the χ2 test, and (3) the behavior of the AFI is highly…”
BackgroundResultInfluentialOn making causal claims: A review and recommendations
Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010) · The Leadership Quarterly · doi:10.1016/j.leaqua.2010.10.010 · Cited by 2,441
How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power
Muthén, L. K., & Muthén, B. O. (2002) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0904_8 · Cited by 2,362
Reporting practices in confirmatory factor analysis: an overview and some recommendations
Jackson, D. L., Gillaspy, J. A., & Purc-Stephenson, R. (2009) · Psychological Methods · doi:10.1037/a0014694 · Cited by 2,242 · cited 2×
“…method, (5) the failure to recognize the existence of equivalent models, (6) poor justification of causal inferences, and (7) selective reporting and the overall lack of clarity in reporting (Breckler, 1990; DiStefano and Hess, 2005; Guo et al., 2009; Jackson et al., 2009; Nunkoo et al., 2013).”
BackgroundMethodThe theoretical status of latent variables.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003) · Psychological Review · doi:10.1037/0033-295x.110.2.203 · Cited by 1,566
Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005) · Journal of Business Research · doi:10.1016/j.jbusres.2003.10.007 · Cited by 906 · cited 2×
“…universal cut-off criteria regardless of the character of the model tested (Marsh et al., 2004; Beauducel and Wittmann, 2005; Fan and Sivo, 2005; Sharma et al., 2005; Yuan, 2005); (2) with several frequently used AFI, the probability of correctly rejecting misspecified models systematically…”
MethodResultFit Indices Versus Test Statistics
Yuan, K.-H. (2005) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr4001_5 · Cited by 415 · cited 2×
“Their distributions (except for RMSEA) are unknown (Yuan, 2005), and although most of them (the group of absolute fit indices) are, apart from sample size and df, a function of the χ2, these indices have no established statistical basis and do not represent a formal test of the model (Hayduk et…”
BackgroundMethodHeene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011) · Psychological Methods · doi:10.1037/a0024917 · Cited by 406 · cited 2×
“These include high power with rising N, rising reliability of indicators and higher communalities (Miles and Shevlin, 2007; Heene et al., 2011), sensitivity to distributional properties of observed variables (Fouladi, 2000) or inflated Type I error rate with rising complexity of the model as a…”
BackgroundMethodAssessment of Fit in Overidentified Models with Latent Variables
WHEATON, B. (1987) · Sociological Methods & Research · doi:10.1177/0049124187016001005 · Cited by 357 · cited 1×
“One fifth of the analyzed studies based their decision to accept the model also on χ2/df although the author of this index withdrew his recommendation for its usage quite a long time ago (see Wheaton, 1987).”
Background
Cited by
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
Swami, V., & Barron, D. (2019) · Body Image · doi:10.1016/j.bodyim.2018.08.014 · Cited by 454
Landerl, K., Freudenthaler, H. H., Heene, M., De Jong, P. F., Desrochers, A., Manolitsis, G., Parrila, R., & Georgiou, G. K. (2019) · Scientific Studies of Reading · doi:10.1080/10888438.2018.1510936 · Cited by 320
The Student Expectations of Learning Analytics Questionnaire
Whitelock‐Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y., & Bennett, K. (2019) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12366 · Cited by 63
“Nevertheless, for both scales, the χ 2 test has found to be significant, which should not be ignored (Ropovik, 2015).”
InfluentialRogers, A. H., Garey, L., Bakhshaie, J., Viana, A. G., Ditre, J. W., & Zvolensky, M. J. (2020) · The Clinical Journal of Pain · doi:10.1097/ajp.0000000000000870 · Cited by 19
Metadata
- Authors
- Ropovik, I.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2015-11-06
- Volume
- 6
- Pages
- 1715-1715
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2118829873
- Cited by
- 116
- References
- 12
- FWCI
- 8.73
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.