CoChat
Sign Up
Cited by 17Psychometric Methodologies and TestingSurvey Methodology and NonresponsePsychological Testing and Assessment

Detecting careless responding in multidimensional forced-choice questionnaires

Open access
Kupffer, Rebekka, Frick, Susanne, Wetzel, Eunike·2024·Educational and Psychological Measurement
DOI 10.1177/00131644231222420PMC PMC11418602
Open this paper in CoChat — free account

Abstract

The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and additionally develop several new indices that are unique to the MFC format. We applied these indices to a data set from an online survey ( N = 1,169) that included a series of personality questionnaires in the MFC format. The correlations among the careless responding indices were somewhat lower than those published for rating scales. Results from a latent profile analysis suggested that the majority of the sample (about 76–84%) did not respond carelessly, although the ones who did were characterized by different levels of careless responding. In a simulation study, we simulated different careless responding patterns and varied the overall proportion of carelessness in the samples. With one exception, the indices worked as intended conceptually. Taken together, the results suggest that careless responding also plays an important role in the MFC format. Recommendations on how it can be addressed are discussed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/00131644231222420

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is no unified recommended threshold for the longstring index.

    “There is no unified recommended threshold on the longstring index”

    From Measuring undergraduate students’ reliance on generative AI during problem-solving: Scale development and validation. (2025) · cites this paper as Kupffer et al., 2024

    The Results section says there were no established cutoff values for the newly developed MFC indices, including longOrderMax and longOrderAvg. The Rank Order 2 Indices section distinguishes those indices from the LongString index used with rating scales. The paper therefore supports the point for its MFC adaptations, but does not establish that no unified threshold exists for the LongString index generally. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. Abbey J. D. Meloy M. G. (2017). Attention by design: Using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management, 53–56(1), 63– 70. 10.1016/j.jom.2017.06.001 · DOI · Paper page
  2. Arias V. B. Garrido L. E. Jenaro C. Martínez-Molina A. Arias B. (2020). A little garbage in, lots of garbage out: Assessing the impact of careless responding in personality survey data. Behavior Research Methods, 52(6), 2489– 2505. 10.3758/s13428-020-01401-8 32462604 · DOI
  3. Ashton M. C. Lee K. (2009). The HEXACO-60: A short measure of the major dimensions of personality. Journal of Personality Assessment, 91(4), 340– 345. 10.1080/00223890902935878 20017063 · DOI
  4. Bartlett M. S. (1950). Tests of significance in factor analysis. British Journal of Statistical Psychology, 3(2), 77– 85. 10.1111/j.2044-8317.1950.tb00285.x · DOI
  5. Birkeland S. A. Manson T. M. Kisamore J. L. Brannick M. T. Smith M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14(4), 317– 335. 10.1111/j.1468-2389.2006.00354.x · DOI
  6. Bowling N. A. Gibson A. M. Houpt J. W. Brower C. K. (2021). Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718– 738. 10.1177/1094428120947794 · DOI
  7. Bowling N. A. Huang J. L. Bragg C. B. Khazon S. Liu M. Blackmore C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218– 229. 10.1037/pspp0000085 26927958 · DOI
  8. Brown A. (2016). Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1), 135– 160. 10.1007/s11336-014-9434-9 25663304 · DOI
  9. Brown A. Maydeu-Olivares A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460– 502. 10.1177/0013164410375112 · DOI
  10. Brown A. Maydeu-Olivares A. (2018). Modeling of forced-choice response formats. In Irwing P. Booth T. Hughes D. (Eds.), The Wiley handbook of psychometric testing (pp. 523– 569). John Wiley & Sons.
  11. Cao M. Drasgow F. (2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347– 1368. 10.1037/apl0000414 31070382 · DOI
  12. Christiansen N. D. Burns G. N. Montgomery G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267– 307. 10.1207/s15327043hup1803_4 · DOI · Paper page
  13. Clark M. E. Gironda R. J. Young R. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15(2), 223– 234. 10.1037/1040-3590.15.2.223 12847783 · DOI
  14. Credé M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596– 612. 10.1177/0013164410366686 · DOI
  15. DeSimone J. A. DeSimone A. J. Harms P. D. Wood D. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology, 67(2), 309– 338. 10.1111/apps.12117 · DOI
  16. DeSimone J. A. Harms P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33(5), 559– 577. 10.1007/s10869-017-9514-9 · DOI
  17. DeSimone J. A. Harms P. D. DeSimone A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171– 181. 10.1002/job.1962 · DOI
  18. Dupuis M. Meier E. Cuneo F. (2019). Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices. Behavior Research Methods, 51(5), 2228– 2237. 10.3758/s13428-018-1103-y 30091086 · DOI
  19. Eid M. Gollwitzer M. Schmitt M. (2017). Statistik und Forschungsmethoden: Lehrbuch [Statistics and research methods]. (Originalausgabe, 5., korrigierte edition). Beltz.
  20. Frick S. (2023). Estimating and Using Block Information in the Thurstonian IRT Model. Psychometrika, 88(4), 1556– 1589. 10.1007/s11336-023-09931-8 37640828 PMC10656335 · DOI
  21. Galesic M. Bosnjak M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349– 360. 10.1093/poq/nfp031 · DOI
  22. Goldberg L. R. Johnson J. A. Eber H. W. Hogan R. Ashton M. C. Cloninger C. R. Gough H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40(1), 84– 96. 10.1016/j.jrp.2005.08.007 · DOI · Paper page
  23. Guenole N. Brown A. A. Cooper A. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513– 526. 10.1177/1073191116641181 27056730 · DOI
  24. Hallquist M. N. Wiley J. F. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621– 638. 10.1080/10705511.2017.1402334 30083048 PMC6075832 · DOI
  25. Hicks L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167– 184. 10.1037/h0029780 · DOI
  26. Huang J. L. Curran P. G. Keeney J. Poposki E. M. DeShon R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99– 114. 10.1007/s10869-011-9231-8 · DOI
  27. Huang J. L. Liu M. Bowling N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828– 845. 10.1037/a0038510 25495093 · DOI
  28. Johnson J. A. (2005). Ascertaining the validity of individual protocols from Web-based personality inventories. Journal of Research in Personality, 39(1), 103– 129. 10.1016/j.jrp.2004.09.009 · DOI
  29. Jones D. N. Paulhus D. L. (2014). Introducing the Short Dark Triad (SD3): A brief measure of dark personality traits. Assessment, 21(1), 28– 41. 10.1177/1073191113514105 24322012 · DOI · Paper page
  30. Kaiser H. F. Rice J. (1974). Little Jiffy, Mark Iv. Educational and Psychological Measurement, 34(1), 111– 117. 10.1177/001316447403400115 · DOI · Paper page
  31. Kam C. C. S. Meyer J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512– 541. 10.1177/1094428115571894 · DOI
  32. Kupffer R. Frick S. Wetzel E. (2022b). R package: Detecting careless responding in multidimensional forced-choice questionnaires [Computer software]. https://github.com/rkupffer/CRinMFC PMC11418602 39318479
  33. Lee P. Joo S.-H. Stark S. (2021). Detecting DIF in Multidimensional Forced Choice Measures Using the Thurstonian Item Response Theory Model. Organizational Research Methods, 24(4), 739– 771. 10.1177/1094428120959822 · DOI
  34. Maniaci M. R. Rogge R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61– 83. 10.1016/j.jrp.2013.09.008 · DOI · Paper page
  35. Maydeu-Olivares A. Brown A. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45(6), 935– 974. 10.1080/00273171.2010.531231 26760724 · DOI
  36. Meade A. W. Craig S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437– 455. 10.1037/a0028085 22506584 · DOI · Paper page
  37. Nichols D. S. Greene R. L. Schmolck P. (1989). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: Rationale, development, and empirical trials. Journal of Clinical Psychology, 45(2), 239– 250. 10.1002/1097-4679(198903)45:2<239::aid-jclp2270450210>3.0.co;2-1 2723079 · DOI
  38. Ostoic J. A. R. (2020). Algebraic analysis of multiple social networks with multiplex. Journal of Statistical Software, 92(11), 1– 41. 10.18637/jss.v092.i11 · DOI
  39. Paulhus D. L. (1991). Measurement and control of response bias. In Paulhus D. L. (Ed.), Measures of personality and social psychological attitudes (pp. 17– 59). Elsevier. 10.1016/B978-0-12-590241-0.50006-X · DOI
  40. Pavlov G. Shi D. Maydeu-Olivares A. Fairchild A. (2021). Item desirability matching in forced-choice test construction. Personality and Individual Differences, 183, Article 111114. 10.1016/j.paid.2021.111114 · DOI
  41. Peng S. Man K. Veldkamp B. P. Cai Y. Tu D. (2023). A mixture model for random responding behavior in forced-choice noncognitive assessment: Implication and application in organizational research. Organizational Research Methods, 10944281231181642. 10.1177/10944281231181642 · DOI
  42. Pituch K. A. Stevens J. P. (2015). Applied multivariate statistics for the social sciences: Analyses with SAS and IBM’s SPSS. Routledge.
  43. Pozzebon J. A. Visser B. A. Ashton M. C. Lee K. Goldberg L. R. (2010). Psychometric characteristics of a public-domain self-report measure of vocational interests: The Oregon Vocational Interest Scales. Journal of Personality Assessment, 92(2), 168– 174. 10.1080/00223890903510431 20155566 PMC2822996 · DOI
  44. Rios J. A. Guo H. Mao L. Liu O. L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1), 74– 104. 10.1080/15305058.2016.1231193 · DOI
  45. Rosenberg J. Beymer P. Anderson D. van Lissa C. Schmidt J. (2018). TidyLPA: An R package to easily carry out latent profile analysis (LPA) using open-source or commercial software. Journal of Open Source Software, 3(30), Article 978. 10.21105/joss.00978 · DOI
  46. Schmitt N. Stuits D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9(4), 367– 373. 10.1177/014662168500900405 · DOI
  47. Soto C. J. John O. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117– 143. 10.1037/pspp0000096 27055049 · DOI · Paper page
  48. Teitcher J. E. F. Bockting W. O. Bauermeister J. A. Hoefer C. J. Miner M. H. Klitzman R. L. (2015). Detecting, preventing, and responding to “fraudsters” in internet research: Ethics and tradeoffs. The Journal of Law, Medicine & Ethics, 43(1), 116– 133. 10.1111/jlme.12200 PMC4669957 25846043 · DOI
  49. Ulitzsch E. Pohl S. Khorramdel L. Kroehne U. von Davier M. (2022). A response-time-based latent response mixture model for identifying and modeling careless and insufficient effort responding in survey data. Psychometrika, 87(2), 593– 619. 10.1007/s11336-021-09817-7 34855118 PMC9166878 · DOI
  50. van Buuren S. Groothuis-Oudshoorn K. (2011). Mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1– 67. 10.18637/jss.v045.i03 · DOI · Paper page
  51. Viswesvaran C. Ones D. S. (1999). Meta-analyses of Fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59(2), 197– 210. 10.1177/00131649921969802 · DOI
  52. Walton K. E. Cherkasova L. Roberts R. D. (2020). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment, 27(4), 706– 718. 10.1177/1073191119843585 31007043 · DOI
  53. Watrin L. Geiger M. Spengler M. Wilhelm O. (2019). Forced-choice versus likert responses on an occupational big five questionnaire. Journal of Individual Differences, 40(3), 134– 148. 10.1027/1614-0001/a000285 · DOI
  54. Wetzel E. Frick S. (2020). Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format. Psychological Assessment, 32(3), 239– 253. 10.1037/pas0000781 31738070 · DOI
  55. Wetzel E. Frick S. Brown A. (2021). Does multidimensional forced-choice prevent faking? Comparing the susceptibility of the multidimensional forced-choice format and the rating scale format to faking. Psychological Assessment, 33(2), 156– 170. 10.1037/pas0000971 33151727 · DOI
  56. Wetzel E. Frick S. Greiff S. (2020). The multidimensional forced-choice format as an alternative for rating Scales. European Journal of Psychological Assessment, 36(4), 511– 515. 10.1027/1015-5759/a000609 · DOI
  57. Wickham H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag. https://ggplot2.tidyverse.org
  58. Wise S. L. Kong X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163– 183. 10.1207/s15324818ame1802_2 · DOI
  59. Youden W. J. (1950). Index rating for diagnostic tests. Cancer, 3(1), 32– 35. 10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3 15405679 · DOI
  60. Zhang B. Sun T. Drasgow F. Chernyshenko O. S. Nye C. D. Stark S. White L. A. (2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569– 590. 10.1177/1094428119836486 · DOI

Cited by

Sorted by citations

Authors
Kupffer, R., Frick, S., & Wetzel, E.
Affiliations
  • Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (DE)
  • University of Mannheim (DE)
Corresponding author
Kupffer, Rebekka
ORCID
Venue
Educational and Psychological Measurement
Publisher
SAGE Publications
Published
2024-01-12
Volume
84
Issue
5
Pages
887-926
ISSN
  • 0013-1644 (ISSN-L)
  • 1552-3888
DOI
10.1177/00131644231222420
PMC
PMC11418602
OpenAlex
W4390794801
Cited by
17
References
3
Citations by year
2026: 4 · 2025: 10 · 2024: 2 · 2023: 1
FWCI
4.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Educational and Psychological Measurement (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
  • Young Scholar Fund of the University of Konstanz awarded to Eunike Wetzel
  • Deutsche Forschungsgemeinschaft (GRK 2277)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
Keywords
careless responding, forced-choice questionnaires, personality tests, rating scales, latent profile analysis, simulation study, survey data quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review