How IRT can solve problems of ipsative data in forced-choice questionnaires.
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion-related validity (validity coefficients must sum to zero), and reliability estimates. We argue that these problems are caused by inadequate scoring of forced-choice items and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstone's law of comparative judgment. We show that when Thurstonian IRT modeling is applied (Brown & Maydeu-Olivares, 2011), even existing forced-choice questionnaires with challenging features can be scored adequately and that the IRT-estimated scores are free from the problems of ipsative data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The measurement of psychological androgyny
Bem, S. L. (1974) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0036215 · Cited by 8,522
Estimation of Latent Ability Using a Response Pattern of Graded Scores
Samejima, F. (1969) · Psychometrika · doi:10.1007/bf03372160 · Cited by 2,194
Reconsidering forced-choice item formats for applicant personality assessment
Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005) · Human Performance · doi:10.1207/s15327043hup1803_4 · Cited by 250
The Impact of Faking on Employment Tests: Does Forced Choice Offer a Solution?
Jackson, D. N., Wroblewski, V. R., & Ashton, M. C. (2000) · Human Performance · doi:10.1207/s15327043hup1304_3 · Cited by 233
Increasing Validity with Forced-Choice Criterion Measurement Formats
Bartram, D. (2007) · International Journal of Selection and Assessment · doi:10.1111/j.1468-2389.2007.00386.x · Cited by 132
Cited by
Constructing validity: New developments in creating objective measuring instruments.
Clark, L. A., & Watson, D. (2019) · Psychological Assessment · doi:10.1037/pas0000626 · Cited by 1,470
“…measures in personality and clinical assessment, also are making a comeback in the industrial-organizational psychology literature because of advances in statistical modeling techniques that solve problems of ipsative data that previously plagued this format (e.g., Brown & Maydeu-Olivares, 2013).”
BackgroundSung, Y.-T., & Wu, J.-S. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1041-8 · Cited by 403
Lambert, L. S., & Newman, D. A. (2023) · Organizational Research Methods · doi:10.1177/10944281221115374 · Cited by 259
Kreitchmann, R. S., Abad, F. J., Ponsoda, V., Nieto, M. D., & Morillo, D. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02309 · Cited by 269
Evaluating the metacognitive awareness inventory using empirical factor-structure evidence
Harrison, G. M., & Vallin, L. M. (2018) · Metacognition and Learning · doi:10.1007/s11409-017-9176-z · Cited by 182
“Forced-choice formats are typically less prone to the biases inherent in response-scale format questions, and recent work in analyzing ipsative and quasi-ipsative models (Brown and Maydeu-Olivares 2013) may lead to promising new developments in metacognition measurements.”
Background
Metadata
- Authors
- Brown, A., & Maydeu-Olivares, A.
- Venue
- Psychological Methods
- Published
- 2013
- Volume
- 18
- Issue
- 1
- Pages
- 36-52
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2168767193
- Cited by
- 216
- References
- 5
- FWCI
- 8.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Funders
- Catalan Government (SGR 2009 74)
- Spanish Ministry of Education (PSI2009-07726 and PR2010-0252)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.