Fitting Item Response Theory Models to Two Personality Inventories: Issues and Insights
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
The present study compared the fit of several IRT models to two personality assessment instruments. Data from 13,059 individuals responding to the US-English version of the Fifth Edition of the Sixteen Personality Factor Questionnaire (16PF) and 1,770 individuals responding to Goldberg's 50 item Big Five Personality measure were analyzed. Various issues pertaining to the fit of the IRT models to personality data were considered. We examined two of the most popular parametric models designed for dichotomously scored items (i.e., the two- and three-parameter logistic models) and a parametric model for polytomous items (Samejima's graded response model). Also examined were Levine's nonparametric maximum likelihood formula scoring models for dichotomous and polytomous data, which were previously found to provide good fits to several cognitive ability tests (Drasgow, Levine, Tsien, Williams, & Mead, 1995). The two- and three-parameter logistic models fit some scales reasonably well but not others; the graded response model generally did not fit well. The nonparametric formula scoring models provided the best fit of the models considered. Several implications of these findings for personality measurement and personnel selection were described.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788 · cited 1×
“The NFI and CFI indices for some scales were somewhat lower than the recommended .95 level (Hu & Bentler, 1999).”
BackgroundAlternative Ways of Assessing Model Fit
Browne, M. W., & Cudeck, R. (1992) · Sociological Methods & Research · doi:10.1177/0049124192021002005 · Cited by 27,200 · cited 1×
“The RMSEA values were .08 or less for almost all 16PF scales indicating an adequate fit of the one-factor model (Browne & Cudeck, 1993).”
MethodComparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266
Significance tests and goodness of fit in the analysis of covariance structures.
Bentler, P. M., & Bonett, D. G. (1980) · Psychological Bulletin · doi:10.1037//0033-2909.88.3.588 · Cited by 19,023
THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS
BARRICK, M. R., & MOUNT, M. K. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00688.x · Cited by 8,689
Testing Structural Equation Models.
Clogg, C. C., Bollen, K. A., & Long, J. S. (1995) · Social Forces · doi:10.2307/2580595 · Cited by 8,211
Applications of Item Response Theory To Practical Testing Problems
Lord, F. M. (2012) · Routledge eBooks · doi:10.4324/9780203056615 · Cited by 5,007 · cited 3×
“Regardless of mathematical complexity, the use of any model must eventually be justified on the basis of empirical results, and not only on a priori grounds (Lord, 1980).”
BackgroundMethodInfluentialEssentials of Psychological Testing
Rosner, B., & Cronbach, L. J. (1960) · The American Journal of Psychology · doi:10.2307/1419921 · Cited by 4,513
LISREL 7 : a guide to the program and applications
Jöreskog, K. G., & Sörbom, D. (1989) · Medical Entomology and Zoology · Cited by 4,330
Fundamentals of item response theory
(1992) · Choice Reviews Online · doi:10.5860/choice.29-4185 · Cited by 3,414
(1999) · Medical Entomology and Zoology · Cited by 3,151
(1999) · Cited by 2,653
ESTIMATION OF LATENT ABILITY USING A RESPONSE PATTERN OF GRADED SCORES1
Samejima, F. (1968) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1968.tb00153.x · Cited by 2,587 · cited 1×
“Two polytomous models suitable for ordered response categories were selected for the present study: Samejima’s graded response (SGR) model (Samejima, 1969) and the polytomous MFS model (Levine & Willams, 1993).”
MethodPersonality Psychology in Europe
Vera-Bermudez, J., Raad, D. B., de Vries, J., García, A. M. P., Sanchez-Elvita, A., & Heck, G. V. (1997) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 2,462
Statistically based tests for the number of common factors
Steiger, J. H. (1980) · Medical Entomology and Zoology · Cited by 2,413
Handbook of Modern Item Response Theory
(1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6 · Cited by 2,305
Gender differences in personality: A meta-analysis.
Feingold, A. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.3.429 · Cited by 2,240
Coombs, C. H. (1960) · Psychological Review · doi:10.1037/h0047773 · Cited by 1,555 · cited 2×
“Cronbach (1960) was one of the first psychometricians who emphasized the distinction between tests of typical and maximum performance.”
BackgroundPERSONALITY MEASURES AS PREDICTORS OF JOB PERFORMANCE: A META‐ANALYTIC REVIEW
TETT, R. P., JACKSON, D. N., & ROTHSTEIN, M. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00696.x · Cited by 1,481
Bock, R. D. (1972) · Psychometrika · doi:10.1007/bf02291411 · Cited by 1,228 · cited 3×
“Next, BILOG (Mislevy & Bock, 1991) was used to estimate item parameters for the 3PL model for each of the 16PF and Big Five scales. These item parameters were used to create synthetic data sets that were truly unidimensional and contained the same number of simulated examinees as our original samples. As before, inter…”
MethodInfluentialReise, S. P., Widaman, K. F., & Pugh, R. H. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.552 · Cited by 1,224
Öneş, D. S., Viswesvaran, C., & Schmidt, F. L. (1993) · Journal of Applied Psychology · doi:10.1037//0021-9010.78.4.679 · Cited by 1,081
Methodology Review: Assessing Unidimensionality of Tests and ltenls
Hattie, J. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900204 · Cited by 1,013 · cited 1×
“Hattie (1984, 1985) empirically assessed over 30 indices of unidimensionality and found problems with nearly all of them.”
Barrick, M. R., & Mount, M. K. (1993) · Journal of Applied Psychology · doi:10.1037//0021-9010.78.1.111 · Cited by 899
Toward a theory of individual differences and leadership: Understanding the motivation to lead.
Chan, K.-Y., & Drasgow, F. (2001) · Journal of Applied Psychology · doi:10.1037//0021-9010.86.3.481 · Cited by 845
Self-citationHough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.5.581 · Cited by 842
A Nonparametric Approach for Assessing Latent Trait Unidimensionality
Stout, W. (1987) · Psychometrika · doi:10.1007/bf02294821 · Cited by 504 · cited 5×
“MULTIVARIATE BEHAVIORAL RESEARCH 533 Another frequently used method for assessing unidimensionality is Stout’s nonparametric DIMTEST (Stout, 1987).”
MethodInfluentialAn Investigation of the Parallel Analysis Criterion for Determining the Number of Common Factors
Humphreys, L. G., & Montanelli Jr., R. G. (1975) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr1002_5 · Cited by 392
Personality and personality measurement.
Hogan, R. T. (1991) · Cited by 353
INTEGRITY TESTING FOR PERSONNEL SELECTION: AN UPDATE
SACKETT, P. R., BURRIS, L. R., & CALLAHAN, C. (1989) · Personnel Psychology · doi:10.1111/j.1744-6570.1989.tb00666.x · Cited by 323
Relations between measures of typical and maximum job performance.
Sackett, P. R., Zedeck, S., & Fogli, L. (1988) · Journal of Applied Psychology · doi:10.1037//0021-9010.73.3.482 · Cited by 305
A Response Model for Multiple Choice Items
Thissen, D., & Steinberg, L. (1984) · Psychometrika · doi:10.1007/bf02302588 · Cited by 232
Fitting a Polytomous Item Response Model to Likert-Type Data
Muraki, E. (1990) · Applied Psychological Measurement · doi:10.1177/014662169001400106 · Cited by 228 · cited 2×
“A detailed discussion about fitting the SGR model to Likert-type data can be found in Muraki (1990). In the personality domain, the SGR model has been used to address the specificity of item content in the NEO-PI Conscientiousness scale (Schmit & Ryan, 1997) and to model faking on the Assessment of Biographical and”
MethodFitting Polytomous Item Response Theory Models to Multiple-Choice Tests
Drasgow, F., Levine, M. V., Tsien, S., Williams, B., & Mead, A. D. (1995) · Applied Psychological Measurement · doi:10.1177/014662169501900203 · Cited by 218 · cited 6×
“Drasgow et al. (1995) advocated a combination of complementary graphical and statistical methods to evaluate the adequacy of model predictions.”
MethodResultInfluentialSelf-citationModeling faking good on personality items: An item-level analysis.
Zickar, M. J., & Robie, C. (1999) · Journal of Applied Psychology · doi:10.1037//0021-9010.84.4.551 · Cited by 202
An item response theory analysis of the Hare Psychopathy Checklist--Revised.
Cooke, D. J., & Michie, C. (1997) · Psychological Assessment · doi:10.1037//1040-3590.9.1.3 · Cited by 196
Drasgow, F., & Lissak, R. I. (1983) · Journal of Applied Psychology · doi:10.1037//0021-9010.68.3.363 · Cited by 180
Self-citationFitting the Two-Parameter Model to Personality Data
Reise, S. P., & Waller, N. G. (1990) · Applied Psychological Measurement · doi:10.1177/014662169001400105 · Cited by 175 · cited 5×
“For example, we do not know whether the 3PL model is more appropriate than the 2PL model for personality data: Reise and Waller (1990) did not attempt to fit the 3PL model even though they noted that several items had “lower asymptotes” corresponding to the c-parameter in the 3PL model.”
BackgroundMethodInfluentialValidity Issues in the Likert and Thurstone Approaches to Attitude Measurement
Roberts, J. S., Laughlin, J. E., & Wedell, D. H. (1999) · Educational and Psychological Measurement · doi:10.1177/00131649921969811 · Cited by 165
Effects of the testing situation on item responding: Cause for concern.
Stark, S., Chernyshenko, O. S., Chan, K.-Y., Lee, W. C., & Drasgow, F. (2001) · Journal of Applied Psychology · doi:10.1037//0021-9010.86.5.943 · Cited by 164
Self-citationThe Derivation of Some Tests for the Rasch Model from the Multinomial Distribution
Glas, C. A. W. (1988) · Psychometrika · doi:10.1007/bf02294405 · Cited by 160 · cited 2×
“Our use of 2 statistics for pairs and triples of items is similar in spirit to statistics developed by van den Wollenberg (1982) and Glas (1988). Using the Rasch model, these researchers showed that statistics computed from pairs of items are sensitive to violations of local independence.”
MethodDetecting Faking on a Personality Instrument Using Appropriateness Measurement
Zickar, M. J., & Drasgow, F. (1996) · Applied Psychological Measurement · doi:10.1177/014662169602000107 · Cited by 158 · cited 1×
“Zickar and Drasgow (1996) presented some evidence that the SGR model did not fit the ABLE data well, but did not explore that issue in detail.”
Self-citationAn Empirical Study of Various Indices for Determining Unidimensionality
Hattie, J. (1984) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr1901_3 · Cited by 158 · cited 2×
“Hattie (1984, 1985) empirically assessed over 30 indices of unidimensionality and found problems with nearly all of them.”
MethodTwo New Test Statistics for the Rasch Model
van den Wollenberg, A. L. (1982) · Psychometrika · doi:10.1007/bf02296270 · Cited by 150 · cited 1×
“Our use of 2 statistics for pairs and triples of items is similar in spirit to statistics developed by van den Wollenberg (1982) and Glas (1988).”
MethodHarkness, A. R., & McNulty, J. L. (1994) · Cited by 149
Andrich, D. (1996) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1996.tb01093.x · Cited by 136 · cited 1×
“However, several researchers (Andrich, 1996; Roberts et al., 1999) have argued that this assumption might be flawed for attitude items.”
BackgroundThe Application of an Unfolding Model of the PIRT Type to the Measurement of Attitude
Andrich, D. (1988) · Applied Psychological Measurement · doi:10.1177/014662168801200105 · Cited by 136 · cited 2×
“There has been an increasing number of publications suggesting that the Likert approach to scale construction may MULTIVARIATE BEHAVIORAL RESEARCH 557 have limitations for noncognitive items (e.g., Andrich, 1988, Roberts, Laughlin, & Wedell, 1999).”
BackgroundWaller, N. G., Tellegen, A., McDonald, R. P., & Lykken, D. T. (1996) · Journal of Personality · doi:10.1111/j.1467-6494.1996.tb00521.x · Cited by 110 · cited 2×
“The 2PL model has been used extensively with personality data because of its simplicity and some evidence of model-data fit (Cooke & Miche, 1997; Reise & Waller, 1990; Waller et al., 1996).”
MethodA New Family of Models for the Multiple-Choice Item.
Samejima, F. (1979) · doi:10.21236/ada080350 · Cited by 98
Personality at work: Issues and evidence.
Hough, L. M. (1998) · Cited by 85
Rouse, S. V., Finger, M. S., & Butcher, J. N. (1999) · Journal of Personality Assessment · doi:10.1207/s15327752jp720212 · Cited by 81 · cited 1×
“Reise and Waller argued that guessing was not expected on a personality test such as the MPQ. Subsequently, other researchers have suggested that guessing on cognitive ability items may be analogous to faking on personality items (see Rouse et al., 1999).”
BackgroundAn Item Response Theory Evaluation of an English Version of the Trier Personality Inventory (TPI)
Ellis, B. B., Becker, P., & Kimmel, H. D. (1993) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022193242001 · Cited by 66 · cited 1×
“The only example we could find was the Ellis et al. (1993) paper that used this model to evaluate the measurement equivalence of English and German versions of the Trier Personality Inventory.”
MethodLatent Trait Models and Dichotomization of Graded Responses
Jansen, P. G. W., & Roskam, E. E. (1986) · Psychometrika · doi:10.1007/bf02294001 · Cited by 58 · cited 2×
“If dichotomous IRT models are found not to fit polytomous data, the misfit could be an artifact MULTIVARIATE BEHAVIORAL RESEARCH 531 of dichotomization (Jansen & Roskam, 1986).”
BackgroundMethodInfluentialPeterson, N. G., Hough, L. M., Dunnette, M. D., Rosse, R. L., HOUSTON, J. S., TOQUAM, J. L., & Wing, H. (1990) · Personnel Psychology · doi:10.1111/j.1744-6570.1990.tb01558.x · Cited by 53
Structures of psychological distress: Testing confirmatory hierarchical models.
Tanaka, J. S., & Huba, G. J. (1984) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.52.4.719 · Cited by 46
Variance homogeneity in interactive regression: A clarifying note about data transformations.
Bobko, P., & Russell, C. J. (1990) · Journal of Applied Psychology · doi:10.1037/0021-9010.75.1.95 · Cited by 18
Cited by
Handbook of multimethod measurement in psychology.
(2006) · American Psychological Association eBooks · doi:10.1037/11383-000 · Cited by 481
Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.6.1292 · Cited by 437
Self-citationStark, S., Chernyshenko, O. S., & Drasgow, F. (2005) · Applied Psychological Measurement · doi:10.1177/0146621604273988 · Cited by 188
Self-citationA Discussion of Modern Versus Traditional Psychometrics As Applied to Personality Assessment Scales
Reise, S. P., & Henson, J. M. (2003) · Journal of Personality Assessment · doi:10.1207/s15327752jpa8102_01 · Cited by 164
“Therefore, researchers evaluate whether The formal testing of model assumptions, especially the unidimensionality assumption, as well as the statistical evaluation of model-to-data goodness of fit, are major parts of applying IRT models to real data (e.g., Chernyshenko et al., 2001).”
MethodEstablishing Construct Continua in Construct Validation: The Process of Continuum Specification
Tay, L., & Jebb, A. T. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918775707 · Cited by 59
Metadata
- Authors
- Chernyshenko, O. S., Stark, S., Chan, K.-Y., Drasgow, F., & Williams, B.
- Venue
- Multivariate Behavioral Research
- Publisher
- Informa UK Limited
- Published
- 2001-10
- Volume
- 36
- Issue
- 4
- Pages
- 523-562
- ISSN
- 0027-3171 (ISSN-L)
- 1532-7906
- OpenAlex
- W2063664385
- Cited by
- 202
- References
- 56
- FWCI
- 3.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.