The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Using reversed items in Likert scales: A questionable practice
Open accessAbstract
BACKGROUND: The use of positively worded items and reversed forms aims to reduce response bias and is a commonly used practice nowadays. The main goal of this research is to analyze the psychometric implications of the use of positive and reversed items in measurement instruments. METHOD: A sample of 374 participants was tested aged between 18 and 73 (M=33.98; SD=14.12), 62.60% were women. A repeated measures design was used, evaluating the participants with positive, reversed, and combined forms of a self-efficacy test. RESULTS: When combinations of positive and reversed items are used in the same test the reliability of the test is flawed and the unidimensionality of the test is jeopardized by secondary sources of variance. In addition, the variance of the scores is reduced, and the means differ significantly from those in tests in which all items are either positive or reversed, but not combined. CONCLUSIONS: The results of this study present a trade-off between a potential acquiescence bias when items are positively worded and a potential different understanding when combining regular and reversed items in the same test. The specialized literature recommends combining regular and reversed items for controlling for response style bias, but these results caution researchers in using them as well after accounting for the potential effect of linguistic skills and the findings presented in this study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.5.879 · Cited by 82,194
Principles and Practice of Structural Equation Modeling
Tabri, N., & Elliott, C. M. (2012) · Canadian Graduate Journal of Sociology and Criminology · doi:10.15353/cgjsc-rcessc.v1i1.25 · Cited by 38,891
(1998) · Cited by 36,049
Review of Psychometric theory (2nd ed.).
SITGREAVES, R. (1979) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/018882 · Cited by 11,425
Measurement and Control of Response Bias
Paulhus, D. L. (1991) · Measures of Personality and Social Psychological Attitudes · doi:10.1016/b978-0-12-590241-0.50006-x · Cited by 3,598
ESTIMATION OF LATENT ABILITY USING A RESPONSE PATTERN OF GRADED SCORES1
Samejima, F. (1968) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1968.tb00153.x · Cited by 2,587
Handbook of Modern Item Response Theory
(1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6 · Cited by 2,305
Estimation of Latent Ability Using a Response Pattern of Graded Scores
Samejima, F. (1969) · Psychometrika · doi:10.1007/bf03372160 · Cited by 2,194
The theory and practice of item response theory.
Ayala, D. (2009) · Guilford Press eBooks · Cited by 1,812
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 1×
“The standard error of the estimate was .10 (CI = ± .196), confi dence intervals of the items did not overlap in 14 out of the 20 items when comparing regular and combined forms, so statistically signifi cant differences were found in these 14 items (α = .05; Cumming & Finch, 2006).”
The basics of item response theory
Baker, F. B. (1985) · Cited by 1,362
Response Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,255
Testing for Factorial Invariance in the Context of Construct Validation
Dimitrov, D. M. (2010) · Measurement and Evaluation in Counseling and Development · doi:10.1177/0748175610373459 · Cited by 1,171
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1503_5 · Cited by 973
Response Sets and Test Validity
Cronbach, L. J. (1946) · Educational and Psychological Measurement · doi:10.1177/001316444600600405 · Cited by 898
Positive and negative global self-esteem: A substantively meaningful distinction or artifactors?
Marsh, H. W. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.4.810 · Cited by 811
Weijters, B., Cabooter, E., & Schillewaert, N. (2010) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2010.02.004 · Cited by 801
Further Evidence on Response Sets and Test Design
Cronbach, L. J. (1950) · Educational and Psychological Measurement · doi:10.1177/001316445001000101 · Cited by 734
Lane, S. (2015) · Routledge eBooks · doi:10.4324/9780203102961 · Cited by 728
Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain
Sonderen, E. V., Sanderman, R., & Coyne, J. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068967 · Cited by 544
Program FACTOR at 10: Origins, development and future directions
Ferrando, P., & Lorenzo-Seva, U. (2017) · Psicothema · doi:10.7334/psicothema2016.304 · Cited by 534
Developing and Validating Test Items
Haladyna, T. M., & Rodriguez, M. C. (2013) · Routledge eBooks · doi:10.4324/9780203850381 · Cited by 530
Misresponse to Reversed and Negated Items in Surveys: A Review
Weijters, B., & Baumgartner, H. (2012) · Journal of Marketing Research · doi:10.1509/jmr.11.0368 · Cited by 421
Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis
Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 400
Assessing Three Sources of Misresponse to Reversed Likert Items
Swain, S. D., Weathers, D., & Niedrich, R. W. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.1.116 · Cited by 386
Marsh, H. W. (1986) · Developmental Psychology · doi:10.1037//0012-1649.22.1.37 · Cited by 320
Modeling multiple response processes in judgment and choice.
Böckenholt, U. (2012) · Psychological Methods · doi:10.1037/a0028111 · Cited by 185
Wording Effects in Self-Esteem Scales: Methodological Artifact or Response Style?
Horan, P. M., DiStefano, C., & Motl, R. W. (2003) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1003_6 · Cited by 184
The proximity effect: The role of inter-item distance on reverse-item bias
Weijters, B., Geuens, M., & Schillewaert, N. (2009) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2008.09.003 · Cited by 179
The dilemma of combining positive and negative items in scales
Solís Salazar, M. (2015) · Psicothema · doi:10.7334/psicothema2014.266 · Cited by 177
Kam, C. C. S., & Meyer, J. P. (2015) · Organizational Research Methods · doi:10.1177/1094428115571894 · Cited by 176
Psychometric properties of the Strength and Difficulties Questionnaire from five European countries
Essau, C. A., Olaya, B., Anastassiou‐Hadjicharalambous, X., Pauli, G., Gilvarry, C. M., Bray, D. E., O’Callaghan, J., & Ollendick, T. H. (2012) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1364 · Cited by 168
Schizotypal traits and psychotic-like experiences during adolescence: An update
Fonseca‐Pedrero, E., & Debbané, M. (2017) · Psicothema · doi:10.7334/psicothema2016.209 · Cited by 150
DiStefano, C., & Motl, R. W. (2008) · Personality and Individual Differences · doi:10.1016/j.paid.2008.10.020 · Cited by 141
Item Response Models for Forced-Choice Questionnaires: A Common Framework
Brown, A. (2014) · Psychometrika · doi:10.1007/s11336-014-9434-9 · Cited by 133
The Stability of Individual Response StylesPreprint
Weijters, B., Geuens, M., & Schillewaert, N. (2008) · RePEc: Research Papers in Economics · Cited by 122
Savalei, V., & Falk, C. F. (2014) · Multivariate Behavioral Research · doi:10.1080/00273171.2014.931800 · Cited by 120
Khorramdel, L., & von Davier, M. (2014) · Multivariate Behavioral Research · doi:10.1080/00273171.2013.866536 · Cited by 116
Byrne, B. M., & van de Vijver, F. J. R. (2017) · Psicothema · doi:10.7334/psicothema2017.178 · Cited by 108
New Guidelines for Developing Multiple-Choice Items
Moreno, R., Martínez, R., & Muñiz, J. (2006) · Methodology · doi:10.1027/1614-2241.2.2.65 · Cited by 99
Self-citationCarlson, M., Wilcox, R., Chou, C.-P., Chang, M., Yang, F., Blanchard, J., Marterella, A., Kuo, A., & Clark, F. (2011) · Psychological Assessment · doi:10.1037/a0022484 · Cited by 98
The Role of Language Comprehension in Mathematics and Problem Solving
Suina, J. H. (2013) · Routledge eBooks · doi:10.4324/9780203056820-17 · Cited by 92
Measurement and control of response styles using anchoring vignettes: A model-based approach.
Bolt, D. M., Lu, Y., & Kim, J. (2014) · Psychological Methods · doi:10.1037/met0000016 · Cited by 89
Ebesutani, C. K., Drescher, C. F., Reise, S. P., Heiden, L. J., Hight, T. L., Damon, J. D., & Young, J. N. (2012) · Journal of Personality Assessment · doi:10.1080/00223891.2012.662188 · Cited by 77
Question Wording Effects in the Assessment of Global Self-Esteem
Dunbar, M., Ford, G., Hunt, K., & Der, G. (2000) · European Journal of Psychological Assessment · doi:10.1027//1015-5759.16.1.13 · Cited by 69
How response bias affects the factorial structure of personality self-reports
Navarro-González, D., Lorenzo‐Seva, U., & Vigil‐Colet, A. (2016) · Psicothema · doi:10.7334/psicothema2016.113 · Cited by 66
Acquiescence as a source of bias and model and person misfit: A theoretical and empirical analysis
Ferrando, P. J., & Lorenzo‐Seva, U. (2009) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711009x470740 · Cited by 61
Reversed items in Likert scales: Filtering out invalid responders
Józsa, K., & Morgan, G. A. (2017) · SZTE Publicatio Repozitórium (University of Szeged) · Cited by 59
The Impact of Incorrect Responses to Reverse-Coded Survey Items.
Hughes, G. (2009) · Research in the schools · Cited by 50
Guidelines based on validity criteria for the development of multiple choice items
Moreno, R., Martínez, R., & Muñiz, J. (2015) · Psicothema · doi:10.7334/psicothema2015.110 · Cited by 49
Self-citationAlessandri, G., Vecchione, M., Fagnani, C., Bentler, P. M., Barbaranelli, C., Medda, E., Nisticò, L., Stazi, M. A., & Caprara, G. V. (2010) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705511.2010.510064 · Cited by 46
Ferrando, P. J., Lorenzo‐Seva, U., & Chico, E. (2003) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3803_04 · Cited by 38
The Effects of Vignette Scoring on Reliability and Validity of Self-Reports
von Davier, M., Shin, H. J., Khorramdel, L., et al. (2017) · Applied Psychological Measurement · doi:10.1177/0146621617730389 · Cited by 38
Enterprising personality profile in youth: Components and assessment
Muñiz, J., Suárez‐Álvarez, J., Pedrosa, I., Fonseca‐Pedrero, E., & Garcı́a-Cueto, E. (2014) · Psicothema · doi:10.7334/psicothema2014.182 · Cited by 36
Self-citationNegatively-Worded Multiple Choice Questions: An Avoidable Threat to Validity
Chiavaroli, N. G. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/ca7y-mm27 · Cited by 32
[Imputation methods for missing data in educational diagnostic evaluation].
Fernández-Alonso, R., Suárez‐Álvarez, J., & Muñiz, J. (2012) · PubMed · Cited by 30
Self-citationDavies, M. F. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.4.736 · Cited by 29
A computerized adaptive test for enterprising personality assessment in youth
Pedrosa, I., Suárez‐Álvarez, J., Garcı́a-Cueto, E., & Muñiz, J. (2016) · Psicothema · doi:10.7334/psicothema2016.68 · Cited by 16
Self-citationReverse items are NOT opposites of straightforward items
Chiorri, C., Anselmi, P., & Robusto, E. (2009) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 6
Cited by
Factor structure and validity of the Revised Suicide Crisis Inventory (SCI-2) among Indian adults
Menon, V., Bafna, A. R., Rogers, M. L., Richards, J., & Galynker, I. (2022) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2022.103119 · Cited by 29
Problematic Social Network Use: Structure and Assessment
González-Nuevo, C., Cuesta, M., Postigo, Á., Menéndez-Aller, Á., & Muñiz, J. (2023) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-021-00711-y · Cited by 22
Self-citationPsychometric Properties of the Stress Mindset Measure (SMM) in a Greek Sample.
Karampas, K., Pezirkianidis, C., & Stalikas, A. (2020) · Psychology · doi:10.4236/psych.2020.118079 · Cited by 19
Metadata
- Authors
- Suárez-Álvarez, J., Pedrosa, I., Lozano, L., García-Cueto, E., Cuesta, M., & Muñiz, J.
- Venue
- Psicothema
- Publisher
- Grupo Editorial Psicofundación
- Published
- 2018-05-01
- Volume
- 2
- Issue
- 30
- Pages
- 149-158
- ISSN
- 0214-9915 (ISSN-L)
- 1886-144X
- OpenAlex
- W2883808623
- Cited by
- 422
- References
- 59
- FWCI
- 22.79
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.