The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
One Size Doesn’t Fit All: Using Factor Analysis to Gather Validity Evidence When Using Surveys in Your Research
Open accessAbstract
Across all sciences, the quality of measurements is important. Survey measurements are only appropriate for use when researchers have validity evidence within their particular context. Yet, this step is frequently skipped or is not reported in educational research. This article briefly reviews the aspects of validity that researchers should consider when using surveys. It then focuses on factor analysis, a statistical method that can be used to collect an important type of validity evidence. Factor analysis helps researchers explore or confirm the relationships between survey items and identify the total number of dimensions represented on the survey. The essential steps to conduct and interpret a factor analysis are described. This use of factor analysis is illustrated throughout by a validation of Diekman and colleagues’ goal endorsement instrument for use with first-year undergraduate science, technology, engineering, and mathematics students. We provide example data, annotated code, and output for analyses in R, an open-source programming language and software environment for statistical computing. For education researchers using surveys, understanding the theoretical and statistical underpinnings of survey validity is fundamental for implementing rigorous education research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
Tabachnick, B. G., & Fidell, L. S. (1983) · Cited by 80,032
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Making sense of cronbach’s alpha.
Tavakol, M., & Dennick, R. (2011) · International Journal of Medical Education · doi:10.5116/ijme.4dfb.8dfd · Cited by 15,126
The Scree Test For The Number Of Factors
Cattell, R. B. (1966) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr0102_10 · Cited by 13,554 · cited 1×
“This step is recognized as the greatest challenge of an EFA, and the issue has generated a large amount of debate (e.g., Cattell, 1966; Crawford et al., BOX 4.”
BackgroundConstruct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
The standards for educational and psychological testing.
Eignor, D. R. (2013) · APA handbook of testing and assessment in psychology, Vol. 1: Test theory and testing and assessment in industrial and organizational psychology. · doi:10.1037/14047-013 · Cited by 6,988
Marsh, H. W., Hau, K.-T., & Wen, Z. (2004) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1103_2 · Cited by 6,691
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
Introduction to Classical and Modern Test Theory
Crocker, L. M., & Algina, J. J. (1986) · Cited by 5,090
Losh-Hesselbart, S., & Fowler, F. J. (1985) · Journal of the American Statistical Association · doi:10.2307/2288598 · Cited by 4,764
Messick, S. (1995) · American Psychologist · doi:10.1037/0003-066x.50.9.741 · Cited by 4,198
Scale Development Research: A Content Analysis and Recommendations for Best Practices
Worthington, R. L., & Whittaker, T. A. (2006) · The Counseling Psychologist · doi:10.1177/0011000006288127 · Cited by 4,087
Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013) · Educational and Psychological Measurement · doi:10.1177/0013164413495237 · Cited by 3,931
Practical Assessment, Research & Evaluation, 2000-2001.
Rudner, L. M., & Schafer, W. D. (2001) · Practical assessment, research & evaluation · Cited by 3,905
A First Course in Factor Analysis
Price, B. (1993) · Technometrics · doi:10.1080/00401706.1993.10485363 · Cited by 3,512
A Beginner’s Guide to Factor Analysis: Focusing on Exploratory Factor Analysis
Yong, A. G., & Pearce, S. (2013) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.09.2.p079 · Cited by 3,304
Review of Using Multivariate Statistics.
Hays, W. L. (1983) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/022267 · Cited by 3,139
On the use, the misuse, and the very limited usefulness of Cronbach’s alpha
Sijtsma, K. (2009) · Psychometrika · doi:10.1007/s11336-008-9101-0 · Cited by 2,814
Uses and abuses of coefficient alpha
Schmitt, N. (1996) · Psychological Assessment · doi:10.1037/1040-3590.8.4.350 · Cited by 2,588
Thanks coefficient alpha, we’ll take it from here.
McNeish, D. (2018) · Psychological Methods · doi:10.1037/met0000144 · Cited by 2,533
Nonnormaland Categorical Data in Structural Equation Modeling
Finney, S. J., & DiStefano, C. (2013) · Structural Equation Modeling · doi:10.1108/978-1-62396-246-320251015 · Cited by 1,855
The development of achievement task values: A theoretical analysis
Wigfield, A., & Eccles, J. S. (1992) · Developmental Review · doi:10.1016/0273-2297(92)90011-p · Cited by 1,717
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1061 · Cited by 1,674
Understanding and Using Factor Scores: Considerations for the Applied Researcher
DiStefano, C., Zhu, M., & Mîndrilă, D. (2020) · doi:10.7275/da8t-4g52 · Cited by 1,614
The Developmental Social Psychology of Gender
(2012) · Psychology Press eBooks · doi:10.4324/9781410605245 · Cited by 1,602
Prentice, D. A., & Carranza, E. (2002) · Psychology of Women Quarterly · doi:10.1111/1471-6402.t01-1-00066 · Cited by 1,501
Constructing validity: New developments in creating objective measuring instruments.
Clark, L. A., & Watson, D. (2019) · Psychological Assessment · doi:10.1037/pas0000626 · Cited by 1,470
How to factor-analyze your data right: do’s, don’ts, and how-to’s.
Matsunaga, M. (2010) · International Journal of Psychological Research · doi:10.21500/20112084.854 · Cited by 1,419
Fabrigar, L. R., & Wegener, D. T. (2011) · doi:10.1093/acprof:osobl/9780199734177.001.0001 · Cited by 1,408
A First Course in Factor Analysis 2nd Ed
Comrey, A. L., & Lee, H. B. (1973) · Cited by 1,367
Stephens, N. M., Fryberg, S. A., Markus, H. R., Johnson, C. S., & Covarrubias, R. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0027143 · Cited by 1,312
Men and things, women and people: A meta-analysis of sex differences in interests.
Su, R., Rounds, J., & Armstrong, P. I. (2009) · Psychological Bulletin · doi:10.1037/a0017364 · Cited by 1,187
Diekman, A. B., Brown, E. R., Johnston, A. M., & Clark, E. K. (2010) · Psychological Science · doi:10.1177/0956797610377342 · Cited by 878
(1982) · Cited by 842
(2014) · Research Methods for Engineers · doi:10.1017/cbo9781139542326.008 · Cited by 792
An Introduction to Applied Multivariate Analysis
Raykov, T., & Marcoulides, G. A. (2008) · doi:10.4324/9780203809532 · Cited by 716 · cited 4×
“…the participant’s responses on different subsets of items relate more closely to one another than to other subsets, that is, it is an analysis of the dimensionality among the items (Raykov and Marcoulides, 2008; Leandre et al., 2012; Tabachnick and Fidell, 2013; Kline, 2016; Bandalos, 2018).”
BackgroundMethodInfluentialDunn, K. E., & Mulvenon, S. W. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/jg4h-rb87 · Cited by 707
Ledesma, R. D., & Valero‐Mora, P. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/wjnc-nm63 · Cited by 663
Armbruster, P., Patel, M., Johnson, E., & Weiss, M. (2009) · CBE—Life Sciences Education · doi:10.1187/cbe.09-03-0025 · Cited by 621
Getting Under the Hood: How and for Whom Does Increasing Course Structure Work?
Eddy, S. L., & Hogan, K. A. (2014) · CBE—Life Sciences Education · doi:10.1187/cbe.14-03-0050 · Cited by 553 · cited 1×
“Generally, each question on these instruments asked about something different and did not involve extensive use of measures of validity to ensure that researchers were, in fact, measuring what they intended to measure (Armbruster et al., 2009; Rissing and Cogan, 2009; Eddy and Hogan, 2014).”
BackgroundSelf-citationRuscio, J., & Roche, B. (2012) · Psychological Assessment · doi:10.1037/a0025697 · Cited by 532
Coefficient Alpha: A Reliability Coefficient for the 21st Century?
Yang, Y., & Green, S. B. (2011) · Journal of Psychoeducational Assessment · doi:10.1177/0734282911406668 · Cited by 479
Measurement Model Quality, Sample Size, and Solution Propriety in Confirmatory Factor Models
Gagne, P., & Hancock, G. R. (2006) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr4101_5 · Cited by 397
Measurement Theory and Applications for the Social Sciences
Gagnon, R. J. (2019) · Measurement: Interdisciplinary Research and Perspectives · doi:10.1080/15366367.2019.1610343 · Cited by 393
Non-Graphical Solutions for Cattell’s Scree Test
Raîche, G., Walls, T. A., Magis, D., Riopel, M., & Blais, J.-G. (2013) · Methodology · doi:10.1027/1614-2241/a000051 · Cited by 344
Perry, J. L., Nicholls, A. R., Clough, P. J., & Crust, L. (2015) · Measurement in Physical Education and Exercise Science · doi:10.1080/1091367x.2014.952370 · Cited by 317
Commentary on Coefficient Alpha: A Cautionary Tale
Green, S. B., & Yang, Y. (2009) · Psychometrika · doi:10.1007/s11336-008-9098-4 · Cited by 312
A Suggested Change in Terminology and Emphasis Regarding Validity and Education
Lissitz, R. W., & Samuelsen, K. (2007) · Educational Researcher · doi:10.3102/0013189x07311286 · Cited by 246
Evaluation of Parallel Analysis Methods for Determining the Number of Factors
Crawford, A. V., Green, S. B., Levy, R., Lo, W.-J., Scott, L., Svetina, D., & Thompson, M. S. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164410379332 · Cited by 242
Kane, M. T. (2015) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2015.1060192 · Cited by 238
(2013) · Measurement, Design, and Analysis · doi:10.4324/9780203726389-30 · Cited by 218
Thanks Coefficient Alpha, We Still Need You!
Raykov, T., & Marcoulides, G. A. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164417725127 · Cited by 210
Measurement Theory and Applications for the Social Sciences
Bandalos, D. L. (2018) · Cited by 210
Bandalos, D. L., & Finney, S. J. (2018) · The Reviewer’s Guide to Quantitative Methods in the Social Sciences · doi:10.4324/9781315755649-8 · Cited by 202
Giving back or giving up: Native American student experiences in science and engineering.
Smith, J. L., Cech, E., Metz, A., Huntoon, M., & Moyer, C. (2014) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/a0036945 · Cited by 199
Eddy, S. L., Brownell, S. E., Thummaphan, P., Lan, M.-C., & Wenderoth, M. P. (2015) · CBE—Life Sciences Education · doi:10.1187/cbe.15-05-0108 · Cited by 182
Self-citationA Brief History of the Philosophical Foundations of Exploratory Factor Analysis
Mulaik, S. A. (1987) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr2203_3 · Cited by 180
Reeves, T. D., & Marbach‐Ad, G. (2016) · CBE—Life Sciences Education · doi:10.1187/cbe.15-08-0183 · Cited by 156
ASPECT: A Survey to Assess Student Perspective of Engagement in an Active-Learning Classroom
Wiggins, B. L., Eddy, S. L., Wener-Fligner, L., Freisem, K., Grunspan, D. Z., Theobald, E. J., Timbrook, J., & Crowe, A. J. (2017) · CBE—Life Sciences Education · doi:10.1187/cbe.16-08-0244 · Cited by 125
Self-citationAllen, J. M., Muragishi, G. A., Smith, J. L., Thoman, D. B., & Brown, E. R. (2015) · Translational Issues in Psychological Science · doi:10.1037/tps0000046 · Cited by 124
The Consequences of Consequential Validity
Mehrens, W. A. (1997) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1997.tb00588.x · Cited by 122
Can an Inquiry Approach Improve College Student Learning in a Teaching Laboratory?
Rissing, S. W., & Cogan, J. G. (2009) · CBE—Life Sciences Education · doi:10.1187/cbe.08-05-0023 · Cited by 106
Validating test score meaning and defending test score use: different aims, different methods
Cizek, G. J. (2015) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2015.1063479 · Cited by 66
Watson, J. B. (2004) · J B Lippincott Company eBooks · doi:10.1037/10016-002 · Cited by 59
Wachsmuth, L. P., Runyon, C., Drake, J. M., & Dolan, E. L. (2017) · CBE—Life Sciences Education · doi:10.1187/cbe.16-08-0248 · Cited by 45
Self-citationAndrews, S., Runyon, C., & Aikens, M. L. (2017) · CBE—Life Sciences Education · doi:10.1187/cbe.17-03-0043 · Cited by 38
Self-citationConstruct Validity: Developments and Debates
Slaney, K. L. (2017) · Palgrave Macmillan UK eBooks · doi:10.1057/978-1-137-38523-9_4 · Cited by 3
Cited by
Gender differences in student participation in an active-learning classroom
Aguillon, S. M., Siegmund, G.-F., Petipas, R. H., Drake, A. G., Cotner, S., & Ballen, C. J. (2020) · CBE—Life Sciences Education · doi:10.1187/cbe.19-03-0048 · Cited by 215
Chatzikyriakidou, K., & McCartney, M. (2022) · International Journal of Science Education · doi:10.1080/09500693.2022.2073482 · Cited by 10
Lee, S., Zhong, M., Foster, C., Segura-Totten, M., & McCartney, M. (2022) · Journal of Microbiology & Biology Education · doi:10.1128/jmbe.00126-22 · Cited by 7
“CFA is used to confirm a previously stated theoretical model (33).”
BackgroundMethodInfluentialBayzat, C., & Dinc, L. (2025) · BMC Nursing · doi:10.1186/s12912-025-03503-0 · Cited by 1
Metadata
- Authors
- Knekta, E., Runyon, C., & Eddy, S.
- Venue
- CBE—Life Sciences Education
- Published
- 2019-03
- Volume
- 18
- Issue
- 1
- Pages
- rm1
- ISSN
- 1931-7913 (ISSN-L)
- OpenAlex
- W2919301453
- Cited by
- 630
- References
- 69
- FWCI
- 35.06
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.