Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: We examined the effectiveness of reverse worded items as a means of reducing or preventing response bias. We first distinguished between several types of response bias that are often confused in literature. We next developed arguments why reversing items is probably never a good way to address response bias. We proposed testing whether reverse wording affects response bias with item-level data from the Multidimensional Fatigue Inventory (MFI-20), an instrument that contains reversed worded items. METHODS: With data from 700 respondents, we compared scores on items that were similar with respect either to content or to direction of wording. Psychometric properties of sets of these items worded in the same direction were compared with sets consisting of both straightforward and reversed worded items. RESULTS: We did not find evidence that ten reverse-worded items prevented response bias. Instead, the data suggest scores were contaminated by respondent inattention and confusion. CONCLUSIONS: Using twenty items, balanced for scoring direction, to assess fatigue did not prevent respondents from inattentive or acquiescent answering. Rather, fewer mistakes are made with a 10-item instrument with items posed in the same direction. Such a format is preferable for both epidemiological and clinical studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this context, a study by Sonderen et al. (2013) showed that reverse-worded items may lead to inattention when responding to a questionnaire.”
From Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load (2023) · cites this paper as Sonderen et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“The latter could indicate a measurement issue related to including both positively and negatively phrased items in a scale [111].”
From Validation of the compassionate engagement and action scales, compassion scale, and Sussex-Oxford compassion scales in a French-Canadian sample (2024) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · cited 2×
“We consider this a variety of the Multi-Trait-Multi-Method matrix discussed by Campbell & Fiske [29].”
MethodA three-component conceptualization of organizational commitment
Meyer, J. P., & Allen, N. J. (1991) · Human Resource Management Review · doi:10.1016/1053-4822(91)90011-z · Cited by 11,049 · cited 2×
“1) Meyer and Allen’s [11] scale to assess affective commitment, of which Merritt [12] demonstrated that the answers on reversed items were prone to careless responding and cognitive fatigue.”
BackgroundDevelopment and validation of the Penn State Worry Questionnaire
Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D. (1990) · Behaviour Research and Therapy · doi:10.1016/0005-7967(90)90135-6 · Cited by 5,099 · cited 1×
“’s [14] scale to assess worry contains 11 positively and five reversed worded items, leading Fresco et al.”
BackgroundThe Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734 · cited 2×
“The process of obtaining survey data is complex, with many possibilities of discrepancies arising between the state or opinion the researcher wants to elicit and the answer given by the respondent [3].”
BackgroundSmets, E. M. A., Garssen, B., Bonke, B., & De Haes, J. C. J. M. (1995) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(94)00125-o · Cited by 3,613 · cited 2×
“[28] constructed a version with seven categories, but in accordance with their adaptation in 1996 [27], a version is used with five answer categories, ranging from 1 (‘yes, that is true’) to 5 (‘no, that is not true’).”
BackgroundMethodMeasurement and Control of Response Bias
Paulhus, D. L. (1991) · Measures of Personality and Social Psychological Attitudes · doi:10.1016/b978-0-12-590241-0.50006-x · Cited by 3,598
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852 · cited 1×
“Krosnick [7] mentions a satisficing response style, whereby the respondent deliberately makes less effort to understand all subtleties of the question.”
On the bipolarity of positive and negative affect.
Russell, J. A., & Carroll, J. M. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.1.3 · Cited by 1,481 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundResponse Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,255
Do We Really Need Multiple-Item Measures in Service Research?
Drolet, A. L., & Morrison, D. G. (2001) · Journal of Service Research · doi:10.1177/109467050133001 · Cited by 736 · cited 2×
“As Drolet and Morrison [9] demonstrate, respondents sometimes just answer the first item and assume that this answer also holds for subsequent (considered similar or even identical) items.”
Background
Cited by
Items 501–543 of 543
Allouani, S. A., Zenjari, A., Berrada, A., Amar, M. B., & Zaki, A. (2025) · Journal of Psychology in Africa · doi:10.32604/jpa.2025.070238
“This bias, likely resulting from confusion and inattention caused by the reversed wording of the questionnaires, compromises the reliability of the responses (Sonderen et al., 2013).”
Teaching efficacy: unravelling multidimensionality across diverse educational contexts
Wells, K., Daniels, L. M., & Frenzel, A. (2025) · Current Psychology · doi:10.1007/s12144-025-08407-5
“They found that the general EI factor was a significant positive predictor of happiness and processing, potentially lowering measurement quality (van Sonderen et al., 2013 ).”
Perceptions of complexity in sustainability: Scale construction and validation
Taavila, S., Rutjens, B. T., & Jostmann, N. B. (2025) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2025.102812
Development and validation of the Japanese version of the Pregnancy and Childbirth Questionnaire
YAMAMOTO, M., Haruna, M., Usui, Y., Yonezawa, K., & Tahara‐Sasagawa, E. (2025) · Journal of Japan Academy of Midwifery · doi:10.3418/jjam.jjam-2025-0022
Measuring South Africa destination loyalty levels through recommending and complaining framings
Román, C., Martín, J. C., & du Plessis, E. (2025) · Research in Hospitality Management · doi:10.1080/22243534.2025.2459080
“The controversy surrounding the use of negatively worded questions to control acquiescence bias (Van Sonderen et al., 2013; Kuru & Pasek, 2016; Chyung et al., 2018) could potentially stem from the prospect theory, but this hypothesis has not been studied in the literature.”
Alharbi, F. S., Aljemaiah, A. I., & Osman, M. (2022) · Figshare · doi:10.6084/m9.figshare.19703993
Shamsalinia, A., Moradi, M., Rad, R. E., Ghadimi, R., Farahani, M. A., Masoudi, R., Rabiei, L., & Ghaffari, F. (2021) · Figshare · doi:10.6084/m9.figshare.14234899
Alharbi, F. S., Aljemaiah, A. I., & Osman, M. (2022) · Figshare · doi:10.6084/m9.figshare.19703996
Gázquez‐López, M., García, I. G., González‐García, A., Salvador, A. M., Pérez-Morente, M. Á., Martínez-García, E., & Álvarez-Serrano, M. A. (2024) · Figshare · doi:10.6084/m9.figshare.26606857
Alyami, M. M., Alshammari, K. A., & Alzahrani, H. M. (2025) · Research Square · doi:10.21203/rs.3.rs-7548318/v1
Development and Validation of the New Helping Attitude Scale to Assess Propensity for Serving Others
Trzeciak, S., Mazzarelli, A., Roberts, M. B., & Roberts, B. W. (2025) · Researchers Journal of Internal Medicine · doi:10.63495/3092284 · Cited by 3
Phillips-Beyer, A., Kawata, A. K., Kleinman, L., & Olivieri, A. O. (2025) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-025-00972-4 · Cited by 1
Razaob, N. A., GEOH, M. K., Rasdi, H. F. M., Kadar, M., & Yunus, F. W. (2024) · Malaysian Journal of Medicine and Health Sciences · doi:10.47836/mjmhs20.4.2
Consumer Perception Assessment of Leather Developed Using Discarded Fish Skins
Soni, S., Sharma, A., & Xavier, K. A. M. (2026) · Agricultural Research · doi:10.1007/s40003-026-00958-8
“This approach helps control for response bias and ensures that participants pay attention to each question [ 36, 41, 55].”
Mikura, S., Takada, T., Shimizu, S., & Fukuhara, S. (2026) · Journal of Interprofessional Care · doi:10.1080/13561820.2026.2633619
“To evaluate whether the inclusion of a negatively worded item in the intention to leave increased cognitive load or confusion among respondents, Cronbach’s alpha coefficients were compared between the scale excluding negatively worded items and the full scale (van Sonderen et al., 2013).”
Westerweel, A., Luijten, M. A. J., Grootenhuis, M. A., Terwee, C. B., van Bindsbergen, K. L. A., & Haverman, L. (2026) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-026-01027-y · Cited by 1
“This could partly explain the observed item misfit in this study, particularly for the reversed items, which introduce additional response difficulties [65].”
Obata, M., Iwanaga, Y., Tsuchinaga, M., Mizutani, A., Downes, D. R. J., & Lee, S. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3776734.3794360
Baussard, L. (2018) · theses.fr (ABES) · doi:10.70675/72ec1686ze064z4998z922ezb33f37633857 · Cited by 1
Kindt, M., Visser, R. M., & Peters, J. (2024) · OSF Preprints (OSF Preprints)
Duan, W., Yang, C., Zhang, S., Ma, Z., & Wang, J. (2026) · Research on Social Work Practice · doi:10.1177/10497315261451222
Phan, L. H., Coleman, P. T., & Ojo, V. (2026) · Frontiers in Organizational Psychology · doi:10.3389/forgp.2026.1777594
Analyzing adolescents’ Social Media Posts: Insights from Data Donations in Three European Countries
Schreurs, L., Boeckxstaens, I., Lee, A. Y., Tabruyn, T., Liu, X., Fitzgerald, K., Hancock, J. T., & Vandenbosch, L. (2026) · Mass Communication & Society · doi:10.1080/15205436.2026.2679461 · Cited by 2
“Such effects can arise from methodological artifacts, including respondent inattention or confusion (van Sonderen et al., 2013).”
Circular business model deployment for sustainability performance: a serial mediation model
Ibrahim, A. A., Kumar, A., Shedid, M., & Dutta, S. K. (2026) · Social Responsibility Journal · doi:10.1108/srj-02-2026-0126
Pohl, N. Z., Gottschling, S., Schmidt, H., Kern, D., & Kammerer, Y. (2026) · Discourse Processes · doi:10.1080/0163853x.2026.2689475
“This highlights the popularity of negations for item reversal, which may also be explained by the fact that item wording can largely remain the same (Van Sonderen et al., 2013).”
Xiao, Y., Li, Y., & Liu, Z. (2026) · Educational Psychology · doi:10.1080/01443410.2026.2699810
Mazzini, A., & Riemer, S. (2026) · Royal Society Open Science · doi:10.1098/rsos.252252
Does Learning Statistics Have to Be So Difficult? Evidence from a Randomized Controlled Trial
Williams, H., Kluthe, T., & Stefik, A. (2026) · ACM Transactions on Computing Education · doi:10.1145/3831705
Mersal, F. A., Baqeas, M. H., Alrwili, A. G., Mohammed, B. A., Aldughmi, O. N., Lotfy, E. M., Abdelwahab, A. A., & Elbilgahy, A. A. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1862936
Hür, S., & Kırca, A. Ş. (2026) · Journal of Human Lactation · doi:10.1177/08903344261453690
Hill, R. M., Hendley, T., Starkey, A. G., Trahan, K., & Tucker, R. P. (2026) · Journal of Personality Assessment · doi:10.1080/00223891.2026.2710604
Stein, M., Fruhen, L. S., & Zacher, H. (2026) · Psychological Reports · doi:10.1177/00332941261473850
Upper Secondary School Students’ Academic Wellbeing,– Validating the Academic Wellbeing ScalePreprint
Qvortrup, A., Lykkegaard, E., & Dalbøl, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7345441
Jikihara, Y., Suzumura, S., Hirose, A., & Ando, S. (2026) · Personal Relationships · doi:10.1111/pere.70087
Ireland, M. J., Weerasinghe, S. A., Pallant, R. D., & Marshall, A. P. (2026) · Australian Critical Care · doi:10.1016/j.aucc.2026.101674
Fekonja, U., Strnad, M., & Fekonja, Z. (2026) · Nursing Open · doi:10.1002/nop2.70850
Deindividuation Among Gendered Violence Survivors and Posting About #MeToo
Pappas, S., Coduto, K. D., & Anderson, J. (2026) · Journal of Media Psychology · doi:10.1027/1864-1105/a000521
Li, S., Tang, J., Qiang, Z., & Wang, Z. (2026) · Journal of Intelligence · doi:10.3390/jintelligence14100232
“A reverse-worded IMP item reduced internal consistency substantially ( α = 0.043 versus 0.815 for the validated two-item form), consistent with the item-removal decision in Jiang and Wang (2026) and evidence that reverse wording can introduce method effects (van Sonderen et al., 2013).”
InfluentialAvena, N. M., Kovács, R. E., de la Vega, R., Soraci, P., Somogyi, A., & Szabó, A. (2026) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-026-01731-2
Yang, H. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1922666
“This is consistent with methodological evidence that reverse-worded items often induce interpretation errors, response bias, and systematic measurement error (Sonderen et al., 2013).”
Biçer, B. K., Nayır, T., & Çoşkun, Ö. (2026) · Medical Research Reports · doi:10.55517/mrr.1982527
Sex Positivity Scale: Turkish Adaptation, Validity, and Reliability Study
Kara, Y., & Ünlü, E. P. (2026) · Psikiyatride Guncel Yaklasimlar - Current Approaches in Psychiatry · doi:10.18863/pgy.1971459
Age-inclusive human resource practices and successful aging at work: a mediated moderation model
Lv, X., He, X., & Sui, D. (2026) · Current Psychology · doi:10.1007/s12144-026-10223-4
“HR practices may create supportive conditions that enhance the effectiveness of employees’ proactive work strategies (Bakker & Van Woerkom, 2017 ).”
InfluentialZhou, Z., Zhou, Y., Ouyang, B., Man, S. S., & Chan, A. H. S. (2026) · Healthcare · doi:10.3390/healthcare14131978
Metadata
- Authors
- Sonderen, E. V., Sanderman, R., & Coyne, J. C.
- Affiliations
- University Medical Center Groningen (NL)
- Corresponding author
- Sonderen, Eric van
- ORCID
- Sanderman, Robbert
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-31
- Volume
- 8
- Issue
- 7
- Pages
- e68967
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3729568
- OpenAlex
- W2141632279
- Cited by
- 544
- References
- 29
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 49 · 2023: 48 · 2022: 54 · 2021: 59 · 2020: 70 · 2019: 51 · 2018: 44 · 2017: 27 · 2016: 29 · 2015: 16 · 2014: 3
- FWCI
- 49.55
- Type
- journal article
- Language
- en
- Update notices
- correction (2013-09-13)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Universiteit Maastricht
- Maastricht Universitair Medisch Centrum
- Universitair Medisch Centrum Groningen
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- response bias, acquiescence bias, questionnaire design, Multidimensional Fatigue Inventory, fatigue measurement, psychometric properties
- MeSH terms
- Animals, Cattle, Fatigue, Humans, Psychometrics, Surveys and Questionnaires, Reproducibility of Results, Inflammatory Bowel Diseases, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.