Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: We examined the effectiveness of reverse worded items as a means of reducing or preventing response bias. We first distinguished between several types of response bias that are often confused in literature. We next developed arguments why reversing items is probably never a good way to address response bias. We proposed testing whether reverse wording affects response bias with item-level data from the Multidimensional Fatigue Inventory (MFI-20), an instrument that contains reversed worded items. METHODS: With data from 700 respondents, we compared scores on items that were similar with respect either to content or to direction of wording. Psychometric properties of sets of these items worded in the same direction were compared with sets consisting of both straightforward and reversed worded items. RESULTS: We did not find evidence that ten reverse-worded items prevented response bias. Instead, the data suggest scores were contaminated by respondent inattention and confusion. CONCLUSIONS: Using twenty items, balanced for scoring direction, to assess fatigue did not prevent respondents from inattentive or acquiescent answering. Rather, fewer mistakes are made with a 10-item instrument with items posed in the same direction. Such a format is preferable for both epidemiological and clinical studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this context, a study by Sonderen et al. (2013) showed that reverse-worded items may lead to inattention when responding to a questionnaire.”
From Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load (2023) · cites this paper as Sonderen et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“The latter could indicate a measurement issue related to including both positively and negatively phrased items in a scale [111].”
From Validation of the compassionate engagement and action scales, compassion scale, and Sussex-Oxford compassion scales in a French-Canadian sample (2024) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · cited 2×
“We consider this a variety of the Multi-Trait-Multi-Method matrix discussed by Campbell & Fiske [29].”
MethodA three-component conceptualization of organizational commitment
Meyer, J. P., & Allen, N. J. (1991) · Human Resource Management Review · doi:10.1016/1053-4822(91)90011-z · Cited by 11,049 · cited 2×
“1) Meyer and Allen’s [11] scale to assess affective commitment, of which Merritt [12] demonstrated that the answers on reversed items were prone to careless responding and cognitive fatigue.”
BackgroundDevelopment and validation of the Penn State Worry Questionnaire
Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D. (1990) · Behaviour Research and Therapy · doi:10.1016/0005-7967(90)90135-6 · Cited by 5,099 · cited 1×
“’s [14] scale to assess worry contains 11 positively and five reversed worded items, leading Fresco et al.”
BackgroundThe Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734 · cited 2×
“The process of obtaining survey data is complex, with many possibilities of discrepancies arising between the state or opinion the researcher wants to elicit and the answer given by the respondent [3].”
BackgroundSmets, E. M. A., Garssen, B., Bonke, B., & De Haes, J. C. J. M. (1995) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(94)00125-o · Cited by 3,613 · cited 2×
“[28] constructed a version with seven categories, but in accordance with their adaptation in 1996 [27], a version is used with five answer categories, ranging from 1 (‘yes, that is true’) to 5 (‘no, that is not true’).”
BackgroundMethodMeasurement and Control of Response Bias
Paulhus, D. L. (1991) · Measures of Personality and Social Psychological Attitudes · doi:10.1016/b978-0-12-590241-0.50006-x · Cited by 3,598
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852 · cited 1×
“Krosnick [7] mentions a satisficing response style, whereby the respondent deliberately makes less effort to understand all subtleties of the question.”
On the bipolarity of positive and negative affect.
Russell, J. A., & Carroll, J. M. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.1.3 · Cited by 1,481 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundResponse Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,255
Do We Really Need Multiple-Item Measures in Service Research?
Drolet, A. L., & Morrison, D. G. (2001) · Journal of Service Research · doi:10.1177/109467050133001 · Cited by 736 · cited 2×
“As Drolet and Morrison [9] demonstrate, respondents sometimes just answer the first item and assume that this answer also holds for subsequent (considered similar or even identical) items.”
BackgroundFactors Defined by Negatively Keyed Items: The Result of Careless Respondents?
Schmitt, N., & Stuits, D. M. (1985) · Applied Psychological Measurement · doi:10.1177/014662168500900405 · Cited by 535 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundSmets, E., Garssen, B., Cull, A., & de Haes, J. (1996) · British Journal of Cancer · doi:10.1038/bjc.1996.42 · Cited by 456 · cited 12×
“[28] constructed a version with seven categories, but in accordance with their adaptation in 1996 [27], a version is used with five answer categories, ranging from 1 (‘yes, that is true’) to 5 (‘no, that is not true’).”
MethodInfluentialThe great response-style myth.
Rorer, L. G. (1965) · Psychological Bulletin · doi:10.1037/h0021888 · Cited by 434
Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis
Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 400 · cited 1×
“Woods [19] found that carelessness among 10% of the respondents on a scale containing reversed worded items, would already lead to an artificially created factor, inducing researchers to erroneously reject unidimensionality.”
BackgroundAssessing Three Sources of Misresponse to Reversed Likert Items
Swain, S. D., Weathers, D., & Niedrich, R. W. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.1.116 · Cited by 386 · cited 3×
“Sometimes the respondent may think the item is well understood, but still an error can be made, due to a high level of difficulty [6].”
BackgroundMethodInfluentialReversed item bias: An integrative model.
Weijters, B., Baumgartner, H., & Schillewaert, N. (2013) · Psychological Methods · doi:10.1037/a0032121 · Cited by 386 · cited 1×
“According to Weijters et al. [8] we focus on response style in discussing the consequences of reversed worded items.”
Controlling Acquiescence Response Bias by Item Reversals: The Effect on Questionnaire Validity
Schriesheim, C. A., & Hill, K. D. (1981) · Educational and Psychological Measurement · doi:10.1177/001316448104100420 · Cited by 322 · cited 1×
“Schriesheim and Hill [10] previously concluded that reversed worded items, when used in an effort to control for acquiescence, lowered questionnaires’ validity.”
BackgroundSpector, P. (1997) · Journal of Management · doi:10.1016/s0149-2063(97)90020-9 · Cited by 276 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundWatson, D., & Tellegen, A. (1999) · Psychological Bulletin · doi:10.1037//0033-2909.125.5.601 · Cited by 267
Romberg-Camps, M. J. L., Bol, Y., Dagnelie, P. C., Hesselink-van de Kruijs, M. A. M., Kester, A. D. M., Engels, L. G. J. B., van Deursen, C., Hameeteman, W. H. A., Pierik, M., Wolters, F., et al. (2010) · Inflammatory Bowel Diseases · doi:10.1002/ibd.21285 · Cited by 240 · cited 3×
“In the few patients below 18 years of age their parents or a legal representative signed an informed consent [26].”
BackgroundInfluentialRoszkowski, M. J., & Soven, M. (2010) · Assessment & Evaluation in Higher Education · doi:10.1080/02602930802618344 · Cited by 232 · cited 3×
“In accordance with Roszkowski and Soven [13], we suggest separate presentation of ordinary items and reversed worded items, instead of a list where these items are all mixed up.”
BackgroundMethodInfluentialConfirmatory factor analysis of the Penn State Worry Questionnaire
Fresco, D. M., Heimberg, R. G., Mennin, D. S., & Turk, C. L. (2002) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(00)00113-3 · Cited by 149 · cited 1×
“[15] to the conclusion of two distinct concepts, Worry Engagement and Absence of Worry.”
BackgroundMerritt, S. M. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9252-3 · Cited by 115 · cited 1×
“We mention some examples: 1) Meyer and Allen’s [11] scale to assess affective commitment, of which Merritt [12] demonstrated that the answers on reversed items were prone to careless responding and cognitive fatigue.”
The phoenix of bipolarity: Reply to Watson and Tellegen (1999).
Russell, J. A., & Carroll, J. M. (1999) · Psychological Bulletin · doi:10.1037//0033-2909.125.5.611 · Cited by 111
Factor Structure of the Penn State Worry Questionnaire: Examination of a Method Factor
Hazlett-Stevens, H., Ullman, J. B., & Craske, M. G. (2004) · Assessment · doi:10.1177/1073191104269872 · Cited by 96 · cited 1×
“[16] however conclude that the reversed worded items caused a method factor.”
BackgroundThe acquiescence effect in responding to a questionnaire.
Hinz, A., Michalski, D., Schwarz, R., & Herzberg, P. Y. (2007) · PubMed Central · Cited by 91
How Do Respondents Construe Ambiguous Response Formats of Affect Items?
Segura, S. L., & González-Romá, V. (2003) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.85.5.956 · Cited by 63 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundResponse styles in consumer research
Weijters, B. (2006) · Ghent University Academic Bibliography (Ghent University) · Cited by 37
An Investigation Of 'Honesty Check' Items In Higher Education Course Evaluations
Bradley, K. D., Royal, K. D., & Bradley, J. W. (2011) · Journal of College Teaching & Learning (TLC) · doi:10.19030/tlc.v5i8.1240 · Cited by 12 · cited 1×
“[17] inspected the behavior of ten items from a scale used to ask students to evaluate courses and instructors.”
Method
Cited by
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Muris, P., & Petrocchi, N. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2005 · Cited by 498
Corr, P. J., & Cooper, A. J. (2016) · Psychological Assessment · doi:10.1037/pas0000273 · Cited by 426
Using reversed items in Likert scales: A questionable practice
Suárez-Álvarez, J., Pedrosa, I., Lozano, L., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018) · Psicothema · doi:10.7334/psicothema2018.33 · Cited by 422
“…(SDQ), concluding that when reversed items are removed the model fi t to the data improves signifi cantly, both for the whole sample and by country. van Sonderen et al. (2013) compared the psychometric properties of a set of regular items with a set of items containing both regular and…”
InfluentialA Short Boredom Proneness Scale: Development and Psychometric Properties
Struk, A. A., Carriere, J. S. A., Cheyne, J. A., & Danckert, J. (2017) · Assessment · doi:10.1177/1073191115609996 · Cited by 359
Metadata
- Authors
- Sonderen, E. V., Sanderman, R., & Coyne, J. C.
- Affiliations
- University Medical Center Groningen (NL)
- Corresponding author
- Sonderen, Eric van
- ORCID
- Sanderman, Robbert
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-31
- Volume
- 8
- Issue
- 7
- Pages
- e68967
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3729568
- OpenAlex
- W2141632279
- Cited by
- 544
- References
- 29
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 49 · 2023: 48 · 2022: 54 · 2021: 59 · 2020: 70 · 2019: 51 · 2018: 44 · 2017: 27 · 2016: 29 · 2015: 16 · 2014: 3
- FWCI
- 49.55
- Type
- journal article
- Language
- en
- Update notices
- correction (2013-09-13)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Universiteit Maastricht
- Maastricht Universitair Medisch Centrum
- Universitair Medisch Centrum Groningen
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- response bias, acquiescence bias, questionnaire design, Multidimensional Fatigue Inventory, fatigue measurement, psychometric properties
- MeSH terms
- Animals, Cattle, Fatigue, Humans, Psychometrics, Surveys and Questionnaires, Reproducibility of Results, Inflammatory Bowel Diseases, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.