Criterion-related validity of forced-choice personality measures: A cautionary note regarding Thurstonian IRT versus Classical Test Theory scoring
Open accessAbstract
This study examined criterion-related validity for job-related composites of forced-choice personality scores against job performance using both Thurstonian Item Response Theory (TIRT) and Classical Test Theory (CTT) scoring methods. Correlations were computed across 11 different samples that differed in job or role within a job. A meta-analysis of the correlations (k = 11 and N = 613) found a higher average corrected correlation for CTT (mean ρ = .38) than for TIRT (mean ρ = .00). Implications and directions for future research are discussed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
QIFC is the type of forced-choice inventory whose predictive validity has been most extensively studied.
“in this study we have focused on QIFC inventories because this is the type of FC whose predictive validity has been most extensively studied”
From A Meta-analysis of the Relationship between Cognitive Intelligence and the Big Five Assessed by Quasi-Ipsative Forced-choice Personality Inventories: Implications for Predicting Job Performance (2025) · cites this paper as Fisher et al., 2019
The Measures section describes the inventory as partially ipsative and forced-choice, and Participants and Procedures reports 11 concurrent-validity datasets. These passages show that the paper studies validity for this type of inventory, but they do not establish that its predictive validity has been studied more extensively than that of other forced-choice types. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2010) · Intellectum (Universidad de La Sabana) · doi:10.1111/j.1467- · Cited by 16,131 · see citing works
The development of markers for the Big-Five factor structure
Goldberg, L. R. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.1.26 · Cited by 5,530 · see citing works
Tests for comparing elements of a correlation matrix
Steiger, J. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.87.2.245 · Cited by 4,621 · cited 1× · see citing works
“Criterion-related validity coefficients for the 11 different samples for CTT and TIRT scoring methods are presented in Table 1.2 Tests of dependent correlations (|Z|; Steiger, 1980) were used to compare the correlations across scoring methods.”
MethodInfluentialBetween facets and domains: 10 aspects of the Big Five.
DeYoung, C. G., Quilty, L. C., & Peterson, J. B. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.93.5.880 · Cited by 1,920 · see citing works
Comparative analysis of the reliability of job performance ratings.
Viswesvaran, C., Öneş, D. S., & Schmidt, F. L. (1996) · Journal of Applied Psychology · doi:10.1037/0021-9010.81.5.557 · Cited by 774 · see citing works
Development of a general solution to the problem of validity generalization.
Schmidt, F. L., & Hunter, J. E. (1977) · Journal of Applied Psychology · doi:10.1037/0021-9010.62.5.529 · Cited by 769 · see citing works
The Big Five in personnel selection: Factor structure in applicant and nonapplicant populations.
Schmit, M. J., & Ryan, A. M. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.6.966 · Cited by 422 · see citing works
A Meta‐Analytic Investigation of Job Applicant Faking on Personality Measures
Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. Τ., & Smith, M. A. (2006) · International Journal of Selection and Assessment · doi:10.1111/j.1468-2389.2006.00354.x · Cited by 400 · see citing works
Item Response Modeling of Forced-Choice Questionnaires
Brown, A., & Maydeu‐Olivares, A. (2011) · Educational and Psychological Measurement · doi:10.1177/0013164410375112 · Cited by 333 · see citing works
Reconsidering forced-choice item formats for applicant personality assessment
Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005) · Human Performance · doi:10.1207/s15327043hup1803_4 · Cited by 250 · see citing works
Self-citationHow IRT can solve problems of ipsative data in forced-choice questionnaires.
Brown, A., & Maydeu‐Olivares, A. (2012) · Psychological Methods · doi:10.1037/a0030641 · Cited by 217 · see citing works
Salgado, J. F., & Táuriz, G. (2014) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2012.716198 · Cited by 205 · see citing works
Christiansen, N. D., Goffin, R. D., Johnston, N. G., & Rothstein, M. G. (1994) · Personnel Psychology · doi:10.1111/j.1744-6570.1994.tb01581.x · Cited by 188 · see citing works
Self-citationPsychometric problems and issues involved with creating and using ipsative measures for selection
Meade, A. W. (2004) · Journal of Occupational and Organizational Psychology · doi:10.1348/0963179042596504 · Cited by 168 · see citing works
Heggestad, E. D., Morrison, M., Reeve, C. L. L., & McCloy, R. A. (2006) · Journal of Applied Psychology · doi:10.1037/0021-9010.91.1.9 · Cited by 154 · see citing works
Chernyshenko, O. S., Stark, S. E., Drasgow, F., & Roberts, B. W. (2007) · Psychological Assessment · doi:10.1037/1040-3590.19.1.88 · Cited by 151 · see citing works
Spuriouser and spuriouser: The use of ipsative personality tests
Johnson, C. E., Wood, R. E., & Blinkhorn, S. (1988) · Journal of Occupational Psychology · doi:10.1111/j.2044-8325.1988.tb00279.x · Cited by 148 · see citing works
Salgado, J. F., Anderson, N., & Tauriz, G. (2015) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12098 · Cited by 138 · see citing works
Structural Equation Modeling of Paired-Comparison and Ranking Data.
Maydeu‐Olivares, A., & Böckenholt, U. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.3.285 · Cited by 120 · see citing works
Faking and the validity of conscientiousness: A Monte Carlo investigation.
Komar, S., Brown, D. J., Komar, J. A., & Robie, C. (2008) · Journal of Applied Psychology · doi:10.1037/0021-9010.93.1.140 · Cited by 116 · see citing works
Self-citationFitting a Thurstonian IRT model to forced-choice data using Mplus
Brown, A., & Maydeu‐Olivares, A. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0217-x · Cited by 101 · see citing works
Faking in Personality Assessment: A “Multisaturation” Perspective on Faking as Performance
Tett, R. P., & Simonet, D. V. (2011) · Human Performance · doi:10.1080/08959285.2011.597472 · Cited by 96 · see citing works
On the Statistical and Practical Limitations of Thurstonian IRT Models
Bürkner, P., Schulte, N., & Holling, H. (2019) · Educational and Psychological Measurement · doi:10.1177/0013164419832063 · Cited by 84 · see citing works
Lee, P., Lee, S., & Stark, S. E. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.11.031 · Cited by 48 · see citing works
Further consideration of the use of narrow trait scales.
Christiansen, N. D., & Robie, C. (2011) · Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement · doi:10.1037/a0023069 · Cited by 42 · see citing works
Self-citationForced-choice pre-employment personality assessment: Construct validity and resistance to faking
O’Neill, T., Lewis, R. J., Law, S. J., Larson, N., Hancock, S., Radan, J., Lee, N., & Carswell, J. J. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.03.075 · Cited by 39 · see citing works
ABILITY TO INFLUENCE ONE'S SCORE ON A TYPICAL PENCIL-AND-PAPER TEST OF PERSONALITY *
Kelly, E. L., Miles, C. C., & Terman, L. M. (1936) · Journal of Personality · doi:10.1111/j.1467-6494.1936.tb02123.x · Cited by 39 · see citing works
Dueber, D., Love, A. M. A., Toland, M. D., & Turner, T. A. (2018) · Educational and Psychological Measurement · doi:10.1177/0013164417752782 · Cited by 32 · see citing works
Ling, Y., Zhang, M., Locke, K. D., Li, G., & Li, Z. (2015) · Journal of Personality Assessment · doi:10.1080/00223891.2015.1077852 · Cited by 28 · see citing works
Using IRT Trait Estimates Versus Summated Scores in Predicting Outcomes
Xu, T., & Stone, C. A. (2011) · Educational and Psychological Measurement · doi:10.1177/0013164411419846 · Cited by 25 · see citing works
Ferrando, P. J., & Chico, E. (2007) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 15 · see citing works
Speer, A. B., Robie, C., & Christiansen, N. D. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.058 · Cited by 11 · see citing works
Self-citationOn the Statistical and Practical Limitations of Thurstonian IRT ModelsPreprint
Bürkner, P., Schulte, N., & Holling, H. (2018) · doi:10.31234/osf.io/dbwn8 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis (3rd ed.). Los Angeles, CA: Sage.
- Tett, R.P. & Christiansen, N.D. (2007). Personality tests at the crossroads: A reply to Morgeson, Campion, Dipboye, Hol- lenbeck, Murphy, and Schmitt. Personnel Psychology, 60, 267-293. doi:10.1111/j.1744-6570.2007.00098 · DOI
Cited by
Sorted by citations
Prevalence and Correlates of Academic Dishonesty: Towards a Sustainable University
Cuadrado, D., Salgado, J. F., & Moscoso, S. (2019) · Sustainability · doi:10.3390/su11216062 · Cited by 50 · see citing works
Murano, D. M., Lipnevich, A. A., Walton, K. E., Burrus, J. T., Way, J. D., & Anguiano‐Carrasco, C. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110012 · Cited by 45 · see citing works
Ng, V., Lee, P., Ho, M. R., Kuykendall, L. E., Stark, S. E., & Tay, L. (2020) · Journal of Personality Assessment · doi:10.1080/00223891.2020.1739056 · Cited by 35 · see citing works
Can High-Dimensional Questionnaires Resolve the Ipsativity Issue of Forced-Choice Response Formats?
Schulte, N., Holling, H., & Bürkner, P. (2020) · Educational and Psychological Measurement · doi:10.1177/0013164420934861 · Cited by 34 · see citing works
A Meta-Analysis of the Faking Resistance of Forced-Choice Personality Inventories
Martínez, A., & Salgado, J. F. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.732241 · Cited by 33 · see citing works
Lee, P., Joo, S., & Stark, S. E. (2020) · Organizational Research Methods · doi:10.1177/1094428120959822 · Cited by 25 · see citing works
The Journey of Forced Choice Measurement Over 80 Years: Past, Present, and Future
Lee, P., Son, M., Zhou, S., Joo, S., Jia, Z., & Cheng, V. (2025) · Organizational Research Methods · doi:10.1177/10944281251350687 · Cited by 20 · see citing works
Lee, P., & Joo, S. (2021) · Personnel Assessment and Decisions · doi:10.25035/pad.2021.01.004 · Cited by 18 · see citing works
Hacking the perfect score on high-stakes personality assessments with generative AI
Phillips, J., & Robie, C. (2024) · Personality and Individual Differences · doi:10.1016/j.paid.2024.112840 · Cited by 17 · see citing works
Self-citationEvaluating the Impact of Faking on the Criterion‐Related Validity of Personality Assessments
Speer, A. B., Delacruz, A. Y., Chawota, T. A., Wegmeyer, L. J., Tenbrink, A. P., Gibson, C., & Frost, C. (2025) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12518 · Cited by 17 · see citing works
Self-citationLee, P., Joo, S., Zhou, S., & Son, M. (2022) · Personality and Individual Differences · doi:10.1016/j.paid.2022.111555 · Cited by 16 · see citing works
Joo, S., Lee, P., & Stark, S. E. (2021) · Multivariate Behavioral Research · doi:10.1080/00273171.2021.1960142 · Cited by 16 · see citing works
An updated survey of beliefs and practices related to faking in individual assessments
Robie, C., Risavy, S. D., Jacobs, R. R., Christiansen, N. D., König, C. J., & Speer, A. B. (2021) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12335 · Cited by 13 · see citing works
Self-citationReliability Estimates for IRT-Based Forced-Choice Assessment Scores
Lin, Y. (2021) · Organizational Research Methods · doi:10.1177/1094428121999086 · Cited by 13 · see citing works
Measuring Achievement Striving via a Situational Judgment Test: The Value of Additional Context
Golubovich, J., Lake, C. J., Anguiano‐Carrasco, C., & Seybert, J. M. (2020) · Journal of Work and Organizational Psychology · doi:10.5093/jwop2020a15 · Cited by 12 · see citing works
Speer, A. B., & Delacruz, A. Y. (2021) · International Journal of Selection and Assessment · doi:10.1111/ijsa.12345 · Cited by 12 · see citing works
Self-citationMartínez, A., Salgado, J. F., & Lado, M. (2022) · Revista de Psicología del Trabajo y de las Organizaciones · doi:10.5093/jwop2022a16 · Cited by 11 · see citing works
Contributions to Constructing Forced-Choice Questionnaires Using the Thurstonian IRT Model
Sun, L., Qin, Z., Wang, S., Tian, X., & Luo, F. (2023) · Multivariate Behavioral Research · doi:10.1080/00273171.2023.2248979 · Cited by 5 · see citing works
Zhou, S., Lee, P., & Fyffe, S. (2024) · Human Resource Development Quarterly · doi:10.1002/hrdq.21521 · Cited by 5 · see citing works
Martínez, A., Moscoso, S., & Lado, M. A. (2021) · Sustainability · doi:10.3390/su13084398 · Cited by 3 · see citing works
Robie, C., Feeney, J. R., Rasheed, S., Bourdage, J. S., Christiansen, N. D., Dunlop, P. D., & Daljeet, K. N. (2026) · International Journal of Selection and Assessment · doi:10.1111/ijsa.70075 · Cited by 2 · see citing works
Self-citationMethodological Updates in Personality Assessment for High-Stakes Use
Meade, A. W., & Priest, L. I. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197611050.003.0003 · Cited by 2 · see citing works
Acuña, M. I., Alonso, D., Reyna, C., & Brussino, S. (2022) · Revista Iberoamericana de Diagnóstico y Evaluación - e Avaliação Psicológica · doi:10.21865/ridep64.3.14 · Cited by 2 · see citing works
Can High-Dimensional Questionnaires Resolve the Ipsativity Issue of Forced-Choice Response Formats?Preprint
Schulte, N., Holling, H., & Bürkner, P. (2019) · doi:10.31234/osf.io/43uq8 · Cited by 2 · see citing works
Martínez, A., Lado, M., Cuadrado, D., & Salgado, J. F. (2025) · Journal of Work and Organizational Psychology · doi:10.5093/jwop2025a3 · Cited by 1 · see citing works
Testing the validity of a Latvian multidimensional forced-choice personality inventory
Berga, L., & Austers, I. (2022) · Psihologija · doi:10.2298/psi220307031b · Cited by 1 · see citing works
Robie, C., & Christiansen, N. D. (2021) · Personnel Assessment and Decisions · doi:10.25035/pad.2021.01.001 · Cited by 1 · see citing works
Self-citationHughes, A. W., Dunlop, P. D., Holtrop, D., & Wee, S. (2021) · doi:10.31234/osf.io/r78mb
Integrating forced choice with rapid response measurement
Rasheed, S., Robie, C., Meade, A. W., Christiansen, N. D., Loy, R. W., & Fisher, P. A. (2025) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.70073
Self-citationIt's All Relative: Degree, Causes, and Impact of Ipsativity on Forced‐Choice Personality Tests
Speer, A. B., Tenbrink, A. P., Wegmeyer, L. J., Delacruz, A. Y., & Chawota, T. A. (2026) · International Journal of Selection and Assessment · doi:10.1111/ijsa.70053
Self-citation
- Authors
- Fisher, P., Robie, C., Christiansen, N., Speer, A., & Schneider, L.
- Affiliations
- Wilfrid Laurier University (CA)
- Central Michigan University (US)
- Wayne State University (US)
- Plum.io
- Venue
- Personnel Assessment and Decisions
- Published
- 2019-07
- Volume
- 5
- Issue
- 1
- ISSN
- 2377-8822 (ISSN-L)
- OpenAlex
- W2958621914
- Cited by
- 31
- References
- 30
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 2 · 2023: 2 · 2022: 4 · 2021: 10 · 2020: 5 · 2019: 2
- FWCI
- 3.75
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Personnel Assessment and Decisions (publishedVersion, cc-by)
- ScholarWorks@BGSU (Bowling Green State University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Social Sciences › Psychology › Clinical Psychology › Personality Traits and Psychology
- Keywords
- criterion validity, classical test theory, employee performance, scoring methods, personnel selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
