Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: We examined the effectiveness of reverse worded items as a means of reducing or preventing response bias. We first distinguished between several types of response bias that are often confused in literature. We next developed arguments why reversing items is probably never a good way to address response bias. We proposed testing whether reverse wording affects response bias with item-level data from the Multidimensional Fatigue Inventory (MFI-20), an instrument that contains reversed worded items. METHODS: With data from 700 respondents, we compared scores on items that were similar with respect either to content or to direction of wording. Psychometric properties of sets of these items worded in the same direction were compared with sets consisting of both straightforward and reversed worded items. RESULTS: We did not find evidence that ten reverse-worded items prevented response bias. Instead, the data suggest scores were contaminated by respondent inattention and confusion. CONCLUSIONS: Using twenty items, balanced for scoring direction, to assess fatigue did not prevent respondents from inattentive or acquiescent answering. Rather, fewer mistakes are made with a 10-item instrument with items posed in the same direction. Such a format is preferable for both epidemiological and clinical studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In this context, a study by Sonderen et al. (2013) showed that reverse-worded items may lead to inattention when responding to a questionnaire.”
From Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load (2023) · cites this paper as Sonderen et al., 2013
The cited source says what the claim attributes to it.
In the citing paper
“The latter could indicate a measurement issue related to including both positively and negatively phrased items in a scale [111].”
From Validation of the compassionate engagement and action scales, compassion scale, and Sussex-Oxford compassion scales in a French-Canadian sample (2024) · cites this paper as [111]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · cited 2×
“We consider this a variety of the Multi-Trait-Multi-Method matrix discussed by Campbell & Fiske [29].”
MethodA three-component conceptualization of organizational commitment
Meyer, J. P., & Allen, N. J. (1991) · Human Resource Management Review · doi:10.1016/1053-4822(91)90011-z · Cited by 11,049 · cited 2×
“1) Meyer and Allen’s [11] scale to assess affective commitment, of which Merritt [12] demonstrated that the answers on reversed items were prone to careless responding and cognitive fatigue.”
BackgroundDevelopment and validation of the Penn State Worry Questionnaire
Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D. (1990) · Behaviour Research and Therapy · doi:10.1016/0005-7967(90)90135-6 · Cited by 5,099 · cited 1×
“’s [14] scale to assess worry contains 11 positively and five reversed worded items, leading Fresco et al.”
BackgroundThe Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734 · cited 2×
“The process of obtaining survey data is complex, with many possibilities of discrepancies arising between the state or opinion the researcher wants to elicit and the answer given by the respondent [3].”
BackgroundSmets, E. M. A., Garssen, B., Bonke, B., & De Haes, J. C. J. M. (1995) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(94)00125-o · Cited by 3,613 · cited 2×
“[28] constructed a version with seven categories, but in accordance with their adaptation in 1996 [27], a version is used with five answer categories, ranging from 1 (‘yes, that is true’) to 5 (‘no, that is not true’).”
BackgroundMethodMeasurement and Control of Response Bias
Paulhus, D. L. (1991) · Measures of Personality and Social Psychological Attitudes · doi:10.1016/b978-0-12-590241-0.50006-x · Cited by 3,598
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852 · cited 1×
“Krosnick [7] mentions a satisficing response style, whereby the respondent deliberately makes less effort to understand all subtleties of the question.”
On the bipolarity of positive and negative affect.
Russell, J. A., & Carroll, J. M. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.1.3 · Cited by 1,481 · cited 1×
“More researchers attribute the claim of multidimensionality that is often made in scales with reversed worded items, not to conceptual differences, but to artificial factors resulting from response bias [20–25].”
BackgroundResponse Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,255
Do We Really Need Multiple-Item Measures in Service Research?
Drolet, A. L., & Morrison, D. G. (2001) · Journal of Service Research · doi:10.1177/109467050133001 · Cited by 736 · cited 2×
“As Drolet and Morrison [9] demonstrate, respondents sometimes just answer the first item and assume that this answer also holds for subsequent (considered similar or even identical) items.”
Background
Cited by
Methods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Muris, P., & Petrocchi, N. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2005 · Cited by 498
Corr, P. J., & Cooper, A. J. (2016) · Psychological Assessment · doi:10.1037/pas0000273 · Cited by 426
Using reversed items in Likert scales: A questionable practice
Suárez-Álvarez, J., Pedrosa, I., Lozano, L., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018) · Psicothema · doi:10.7334/psicothema2018.33 · Cited by 422
“…(SDQ), concluding that when reversed items are removed the model fi t to the data improves signifi cantly, both for the whole sample and by country. van Sonderen et al. (2013) compared the psychometric properties of a set of regular items with a set of items containing both regular and…”
InfluentialA Short Boredom Proneness Scale: Development and Psychometric Properties
Struk, A. A., Carriere, J. S. A., Cheyne, J. A., & Danckert, J. (2017) · Assessment · doi:10.1177/1073191115609996 · Cited by 359
The Emotion Regulation Questionnaire: Psychometric properties in general community samples
Preece, D. A., Becerra, R., Robinson, K., & Gross, J. J. (2020) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1564319 · Cited by 237
Preece, D., Becerra, R., Robinson, K., Dandy, J., & Allan, A. (2018) · Personality and Individual Differences · doi:10.1016/j.paid.2018.05.011 · Cited by 221
Examining the Effect of Reverse Worded Items on the Factor Structure of the Need for Cognition Scale
Zhang, X., Noor, R., & Savalei, V. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157795 · Cited by 204
“For instance, Van Sonderen et al.[7] showed that correlations between pairs of items are often higher when they are worded in the same direction than when they measure the same construct.”
BackgroundReise, S. P., Kim, D. S., Mansolf, M., & Widaman, K. F. (2016) · Multivariate Behavioral Research · doi:10.1080/00273171.2016.1243461 · Cited by 181
The dilemma of combining positive and negative items in scales
Solís Salazar, M. (2015) · Psicothema · doi:10.7334/psicothema2014.266 · Cited by 177
Fu, H., Si, L., & Guo, R. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.820777 · Cited by 170
Reopening Openness to Experience: A Network Analysis of Four Openness to Experience Inventories
Christensen, A. P., Cotter, K. N., & Silvia, P. J. (2019) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1467428 · Cited by 165
Kjell, O. N. E., & Diener, E. (2021) · Journal of Personality Assessment · doi:10.1080/00223891.2020.1737093 · Cited by 161
Nilsen, T., & Gustafsson, J. (2016) · IEA research for education · doi:10.1007/978-3-319-41252-8 · Cited by 263
Krieglstein, F., Beege, M., Rey, G. D., Sanchez-Stockhammer, C., & Schneider, S. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09738-0 · Cited by 139
“In sum, it is recommended to formulate all items in the same direction (Sonderen et al., 2013).”
BackgroundBieda, A., Hirschfeld, G., Schönfeld, P., Brailovskaia, J., Zhang, X. C., & Margraf, J. (2017) · Psychological Assessment · doi:10.1037/pas0000353 · Cited by 137
Jelińska, M., & Paradowski, M. B. (2021) · Online Learning · doi:10.24059/olj.v25i1.2492 · Cited by 157
Preece, D. A., Becerra, R., Robinson, K., Dandy, J., & Allan, A. (2018) · Personality and Individual Differences · doi:10.1016/j.paid.2018.07.025 · Cited by 107
Sport Spectatorship and Life Satisfaction: A Multicountry Investigation
Inoue, Y., Sato, M., Filo, K. R., Du, J., & Funk, D. C. (2017) · Journal of Sport Management · doi:10.1123/jsm.2016-0295 · Cited by 114
The scales of general well-being (SGWB)
Longo, Y., Coyne, I. J., & Joseph, S. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.01.005 · Cited by 137
“From a psychometric perspective, reversed items can be confusing to respondents, particularly in long questionnaires (DeVellis, 2012; van Sonderen et al., 2013); they can produce factor structure problems (Schmitt & Stults, 1985; Woods, 2006); they were less effective in measuring well-being…”
BackgroundLiem, C. C. S., Langer, M., Demetriou, A. M., Hiemstra, A. M. F., Wicaksana, A. S., Born, M. P., & König, C. J. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4_9 · Cited by 142
Montero-Marin, J., Kuyken, W., Crane, C., Gu, J., Baer, R., Al-Awamleh, A. A., Akutsu, S., Araya-Véliz, C., Ghorbani, N., Chen, Z. J., et al. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02638 · Cited by 100
Preece, D. A., Becerra, R., Robinson, K., & Dandy, J. (2017) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-017-9634-6 · Cited by 121
Scaling the notion of Islamic spirituality in the workplace
Adawiyah, W. R., & Pramuka, B. A. (2017) · Journal of Management Development · doi:10.1108/jmd-11-2014-0153 · Cited by 143
Matteucci, M. C., Guglielmi, D., & Lauermann, F. (2017) · Social Psychology of Education · doi:10.1007/s11218-017-9369-y · Cited by 134
Self-Confidence Trumps Knowledge: A Cross-Cultural Study of Security Behavior
Sawaya, Y., Sharif, M., Christin, N., Kubota, A., Nakarai, A., & Yamada, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025926 · Cited by 91
“Similarly to previous work [8, 51, 55], we found that reverse-worded questions were the main threat to the validity and reliability of the scales measured by our survey.”
BackgroundResultInfluentialDevelopment and Psychometric Evaluation of the Adaptive Functions of Music Listening Scale
Groarke, J. M., & Hogan, M. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00516 · Cited by 117
The mixed blessing of cyberloafing on innovation performance during the COVID-19 pandemic
Zhong, J., Chen, Y., Yan, J., & Luo, J. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106982 · Cited by 114
“Of the 10-item scale developed by Steger et al. (2012), nine items were retained, while the reverse coded item was excluded because it might lead to respondents’ inattention and confusion, and then lower accuracy, validity and reliability (Schriesheim et al., 1991; Van Sonderen et al., 2013).”
Preece, D. A., Becerra, R., Boyes, M. E., Northcott, C., McGillivray, L., & Hasking, P. A. (2019) · Personality and Individual Differences · doi:10.1016/j.paid.2019.109721 · Cited by 101
The Relations Among School Climate, Instructional Quality, and Achievement Motivation in Mathematics
Scherer, R., & Nilsen, T. (2016) · IEA research for education · doi:10.1007/978-3-319-41252-8_3 · Cited by 118
“Van Sonderen et al. (2013) further pointed out that simply recoding reversely coded items does not solve the issue of the method bias created by negatively worded items; the hope to correct for potential response bias by introducing such items has not been ful fi lled.”
BackgroundIdentifying Milestones in Language Development for Young Children Ages 1 to 6 Years
Visser‐Bochane, M. I., Reijneveld, S. A., Krijnen, W. P., van der Schans, C. P., & Luinge, M. R. (2019) · Academic Pediatrics · doi:10.1016/j.acap.2019.07.003 · Cited by 108
A validated modification of the vaccine hesitancy scale for childhood, influenza and HPV vaccines
Helmkamp, L. J., Szilagyi, P. G., Zimet, G. D., Saville, A. W., Gurfinkel, D., Albertin, C. S., Breck, A., Vangala, S., & Kempe, A. (2021) · Vaccine · doi:10.1016/j.vaccine.2021.02.039 · Cited by 85
Preece, D. A., Becerra, R., Allan, A., Robinson, K., Chen, W., Hasking, P., & Gross, J. J. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110138 · Cited by 76
Four Years in Review: Statistical Practices of Likert Scales in Human-Robot Interaction Studies
Schrum, M. L., Johnson, M., Ghuy, M., & Gombolay, M. C. (2020) · Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3371382.3380739 · Cited by 72
Intuitive eating and food intake in men and women: Results from the Swiss food panel study
Horwath, C. C., Hagmann, D., & Hartmann, C. (2018) · Appetite · doi:10.1016/j.appet.2018.12.036 · Cited by 80
Zhang, C.-Q., Chung, P.-K., & Si, G. (2017) · Journal of Sport and Health Science · doi:10.1016/j.jshs.2015.09.010 · Cited by 69
“van Sonderen et al.43 demonstrated that reverse-worded items did not prevent response bias, but instead scores were contaminated by respondent inattention and confusion.”
BackgroundInfluentialLew, B., Huen, J. M. Y., Yuan, L., Stack, S. J., Maniam, T., Yip, P. S. F., Zhang, J., & Jia, C.-X. (2018) · Religions · doi:10.3390/rel9010015 · Cited by 75
“As regards the reverse-coded items (also known as negatively worded items), researchers have suggested that these items may be a potential source of method bias (e.g., Podsakoff et al. 2003), and the responses to these items could be contaminated by respondent’s inattention and confusion (e.g., Van Sonderen et al. 201…”
BackgroundArli, D. I., Septianto, F., & Chowdhury, R. M. M. I. (2020) · Journal of Business Ethics · doi:10.1007/s10551-019-04414-2 · Cited by 92
Exploring people’s beliefs about the experience of time
Shardlow, J., Lee, R., Hoerl, C., McCormack, T., Burns, P., & Fernandes, A. (2020) · Synthese · doi:10.1007/s11229-020-02749-2 · Cited by 68
Self-efficacy and personal selling: review and examination with an emphasis on sales performance
Peterson, R. A. (2019) · Journal of Personal Selling and Sales Management · doi:10.1080/08853134.2019.1654390 · Cited by 83
“Numerous researchers have written about the potentially deleterious impact of reverse-worded scale items on scale means, variances, and reliabilities (e.g., Barnette 2000; Hartley 2014; van Sonderen, Sanderman, and Coyne 2013; Woods 2006).”
BackgroundLinking brand and competitive advantage: The mediating effect of positioning and market orientation
Rua, O. L., & Santos, C. (2022) · European Research on Management and Business Economics · doi:10.1016/j.iedeen.2021.100194 · Cited by 153
“Responses will be evaluated on a Likert scale, allowing to translate qualitative responses into quantitative data, which is one of the main reasons for their popularity within academia (Lindwall et al., 2012; Rodebaugh et al., 2007; Roszkowski & Soven, 2010; Sonderen et al., 2013).”
An Assessment of the Dimensionality and Factorial Structure of the Revised Paranormal Belief Scale
Drinkwater, K. G., Denovan, A., Dagnall, N., & Parker, A. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01693 · Cited by 77
Jin, H., Kim, S., Love, A., Jin, Y., & Jie, Z. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1012953 · Cited by 143
“Since negatively phrased items may be associated with respondent errors (Sonderen et al., 2013), reversed codes may have contributed to the low factor loadings in.”
BackgroundPanayiotou, M., Black, L., Carmichael-Murphy, P., Qualter, P., & Humphrey, N. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00063-7 · Cited by 62
Alonso, F., Estéban, C., Montoro, L. V., & Serge, A. (2018) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2018.10.032 · Cited by 66
Establishing Construct Continua in Construct Validation: The Process of Continuum Specification
Tay, L., & Jebb, A. T. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918775707 · Cited by 59
The self-regulation for learning online (SRL-O) questionnaire
Broadbent, J., Panadero, E., Lodge, J. M., & Fuller‐Tyszkiewicz, M. (2022) · Metacognition and Learning · doi:10.1007/s11409-022-09319-6 · Cited by 122
“It was decided that all items should be positively worded, as negatively worded items have previously been found to be confusing for participants and create threats to reliability and validity (Chyung et al., 2018; Suárez Álvarez et al., 2018; Van Sonderen et al., 2013).”
BackgroundSaidi, S. S., & Siew, N. M. (2019) · International Journal of Educational Methodology · doi:10.12973/ijem.5.4.651 · Cited by 84
Rodrigues, F. F., Faustino, T., dos Santos, A. B., Teixeira, E., Cid, L., & Monteiro, D. (2021) · International Journal of Sport and Exercise Psychology · doi:10.1080/1612197x.2021.1907766 · Cited by 83
“While the original version contains ten items, we considered only those positively coded (5 items: e.g., “I feel that I have a number of good qualities”) since reverse coded items can contaminate results (Sonderen et al., 2013).”
BackgroundZhang, X., & Savalei, V. (2015) · Educational and Psychological Measurement · doi:10.1177/0013164415596421 · Cited by 79
“The presence of method effects in Likert scales containing PW and RW items can influence the assessment of the factor structure of the scale and distort parameter estimates (DiStefano & Motl, 2006; Sonderen et al., 2013).”
BackgroundMethodBody Image Self-Discrepancy and Depressive Symptoms Among Early Adolescents
Solomon‐Krakus, S., Sabiston, C. M., Brunet, J., Castonguay, A. L., Maximova, K., & Henderson, M. (2016) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2016.08.024 · Cited by 78
“In addition, items of different valence (positive/negative) on a same scale can foster a different type of response and share variance distinct from the concepts that the factors measure [25].”
BackgroundOptimizing the Measurement of Sexism in Political Surveys
Schaffner, B. F. (2021) · Political Analysis · doi:10.1017/pan.2021.6 · Cited by 55
“(Van Sonderen, Sanderman, and Coyne 2013; Zhang, Noor, and Savalei 2016).5 Altogether, the hostile sexism items as a set have a Cronbach’s alpha of 0.86.”
BackgroundInfluentialFrone, M. R. (2018) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2018.05.001 · Cited by 66
Panayiotou, M., Badcock, J. C., Lim, M. H., Banissy, M. J., & Qualter, P. (2022) · Assessment · doi:10.1177/10731911221119533 · Cited by 66
“This is consistent with existing work and might reflect some of the issues inherent to measures with reverse-worded items (e.g., creation of method factors, an increase in the risk of inattention and confusion; Lindwall et al., 2012; van Sonderen et al., 2013).”
ResultTrait Boredom as a Lack of Agency: A Theoretical Model and a New Assessment Tool
Gorelik, D., & Eastwood, J. D. (2023) · Assessment · doi:10.1177/10731911231161780 · Cited by 57
“Although there is some controversy on this matter (e.g., van Sonderen et al., 2013; Zhang et al., 2016), the lack of reverse scored items may mean that the trait variance of the TBS is confounded by response style variance, or the tendency to give stronger or weaker endorsement of self-report items.”
BackgroundThitimajshima, W., Esichaikul, V., & Krairit, D. (2017) · Electronic Markets · doi:10.1007/s12525-017-0256-3 · Cited by 69
“…particularly inattention and acquiescence, this study did not use them because this practice can be vulnerable to response inconsistencies or misresponses, which can reduce the reliability of the scales and result in unexpected factor structures (Van Sonderen et al. 2013; Weijters et al. 2013).”
BackgroundConcerning Trends in Likert Scale Usage in Human-robot Interaction: Towards Improving Best Practices
Schrum, M., Ghuy, M., Hedlund-Botti, E., Natarajan, M., Johnson, M., & Gombolay, M. (2023) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3572784 · Cited by 68
“summing may have an impact on the dimensionality of the scale due to the confusion that reversed items cause [57, 117].”
BackgroundSbeglia, G. C., & Nehm, R. H. (2019) · Science Education · doi:10.1002/sce.21494 · Cited by 56
Saffran, L., Hu, S., Hinnant, A., Scherer, L. D., & Nagel, S. C. (2020) · PLoS ONE · doi:10.1371/journal.pone.0226711 · Cited by 67
Is It Time to Go Positive? Assessing the Positively Worded System Usability Scale (SUS)
Kortum, P., Acemyan, C. Z., & Oswald, F. L. (2021) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720819881556 · Cited by 51
“However, general research on the issue since then has demonstrated that measures with positively and negatively worded questions result in a negligible impact on reducing acqui-escence bias (van Sonderen, Sanderman, & Coyne, 2013).”
BackgroundMota, J., Martins, J., & Onofre, M. (2021) · BMC Public Health · doi:10.1186/s12889-021-12230-5 · Cited by 72
Brunet, J., Gunnell, K. E., Teixeira, P., Sabiston, C. M., & Bélanger, M. (2016) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.2016-0256 · Cited by 50
Some people are attracted sexually to intelligence: A psychometric evaluation of sapiosexuality
Gignac, G. E., Darbyshire, J., & Ooi, M. (2017) · Intelligence · doi:10.1016/j.intell.2017.11.009 · Cited by 57
Corr, P. J., & Krupıć, D. (2017) · Advances in motivation science · doi:10.1016/bs.adms.2017.02.003 · Cited by 63
The Dunn Worry Questionnaire and the Paranoia Worries Questionnaire: new assessments of worry
Freeman, D., Bird, J. C., Loe, B. S., Kingdon, D., Startup, H., Clark, D. M., Ehlers, A., Černis, E., Wingham, G., Evans, N., et al. (2020) · Psychological Medicine · doi:10.1017/s0033291719000588 · Cited by 48
“Five items are reverse worded, a procedure that can be problematic due to participant inattention and confusion (Woods, 2006; Van Sonderen et al., 2013).”
BackgroundImplications of Item Keying and Item Valence for the Investigation of Construct Dimensionality
Seng Kam, C. C., & Meyer, J. P. (2015) · Multivariate Behavioral Research · doi:10.1080/00273171.2015.1022640 · Cited by 47
“Previous authors have suggested excluding reverse-keyed items in psychological instruments because these items contribute to measurement artifacts (e.g., Lindwall et al., 2012; Magazine, Williams, & Williams, 1996; Schriesheim et al., 1991; Schriesheim & Eisenbach, 1995; van Sonderen et al., 2013).”
BackgroundThe Psychometric Assessment of Empathy: Development and Validation of the Perth Empathy Scale
Brett, J. D., Becerra, R., Maybery, M. T., & Preece, D. A. (2022) · Assessment · doi:10.1177/10731911221086987 · Cited by 74
“In line with the recommendations of van Sonderen et al. (2013) and others (e.g., Murphy et al., 2020; Rodebaugh et al., 2007), no items are worded to require reverse-scoring.”
ResultDoolan, E. L., Bryant, R. A., Liddell, B. J., & Nickerson, A. (2017) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2017.04.005 · Cited by 48
Calderón, C., Ferrando, P. J., Lorenzo‐Seva, U., Higuera, O., y Cajal, T. R., Rogado, J., Mut, M., Capote, A. R., Jara, C., & Jiménez‐Fonseca, P. (2019) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2018.11.017 · Cited by 45
Development and initial validation of the Generalized Pliance Questionnaire
Ruiz, F. J., Suárez-Falcón, J. C., Barbero-Rubio, A., & Flórez, C. L. (2019) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.03.003 · Cited by 43
Miller, L. J., Luebke, J. F., & Matiasek, J. (2018) · Zoo Biology · doi:10.1002/zoo.21445 · Cited by 47
Understanding e-commerce customer behaviors to use drone delivery services: A privacy calculus view
Xie, W. C., Chen, C., & Sithipolvanichgul, J. (2022) · Cogent Business & Management · doi:10.1080/23311975.2022.2102791 · Cited by 48
“Research indicates that reverse-worded items failed to prevent response bias and contaminated data due to respondent inattention and confusion (Sonderen et al., 2013).”
BackgroundPrimi, R., Fruyt, F. D., Santos, D. D. D., Antonoplis, S., & John, O. P. (2019) · International Journal of Testing · doi:10.1080/15305058.2019.1673398 · Cited by 53
“Van Sonderen, Sanderman, and Coyne (2013) concluded “An instrument with all items formulated in the same direction and referring to the intended concept (i.e. fatigue or fitness, depression or happiness) is to be preferred” (p. 6).”
BackgroundPeng, J., Yan, S., Strijker, D., Qun, W., Chen, W., & Ma, Z. (2020) · Land Use Policy · doi:10.1016/j.landusepol.2020.104891 · Cited by 52
Nichols, A., & Edlund, J. E. (2020) · International Journal of Social Research Methodology · doi:10.1080/13645579.2020.1719618 · Cited by 50
Fatalism and Well-Being Across Hispanic Cultures: The Social Fatalism Scales (SFS)
Díaz, D., Blanco, A., Bajo, M., & Stavraki, M. (2015) · Social Indicators Research · doi:10.1007/s11205-014-0825-1 · Cited by 36
“Second, some authors argue that reverseworded items do not prevent response bias (acquiescent answering) or inattentive answering and instead can cause confusion (Van Sonderen et al. 2013).”
BackgroundThe Development of Indonesian Teacher Competence Questionnaire
Panggabean, M. S., & Himawan, K. K. (2016) · Journal of Educational Health and Community Psychology · doi:10.12928/jehcp.v5i2.5134 · Cited by 52
Kam, C. C. S., & Fan, X. (2018) · Organizational Research Methods · doi:10.1177/1094428118790371 · Cited by 45
“…items are psychometrically inferior to positively worded ones, and thus it has been suggested that negatively worded items be excluded from surveys (Lindwall et al., 2012; Magazine, Williams, & Williams, 1996; Schriesheim & Eisenbach, 1995; Schriesheim et al., 1991; van Sonderen et al., 2013).”
Fatigue in cancer patients: comparison with the general population and prognostic factors
Hinz, A., Weis, J., Brähler, E., Härter, M., Geue, K., & Ernst, J. (2020) · Supportive Care in Cancer · doi:10.1007/s00520-019-05260-8 · Cited by 38
“Using items of different orientation can reduce the reliability of the scales [23, 24].”
BackgroundMethodHappiness and age in European adults: The moderating role of gross domestic product per capita.
Morgan, J. R., Robinson, O. C., & Thompson, T. (2015) · Psychology and Aging · doi:10.1037/pag0000034 · Cited by 57
Enthusiastic Acts of Evil: The Assessment of Sadistic Personality in Polish and Italian Populations
Kowalski, C. M., Pierro, R. D., Plouffe, R. A., Rogoza, R., & Saklofske, D. H. (2019) · Journal of Personality Assessment · doi:10.1080/00223891.2019.1673760 · Cited by 40
Boukamba, H. K., Oi, T., & Sano, K. (2020) · Journal of Travel Research · doi:10.1177/0047287519895128 · Cited by 48
Help-Seeking Attitudes and Distress Disclosure Among Syrian Refugees in Germany
Schlechter, P., Kamp, S., Wanninger, K., Knausenberger, J., Wagner, U., Wilkinson, P. O., Nohr, L., & Hellmann, J. H. (2020) · The Counseling Psychologist · doi:10.1177/0011000020974934 · Cited by 44
“Additionally, prior research demonstrated that reverseworded items in general often led to inattention and confusion and did not decrease response bias (van Sonderen et al., 2013).”
BackgroundFuchs, L. M., Jacobsen, J., Walther, L., Hahn, E. W., Ta, T. M. T., Bajbouj, M., & von Scheve, C. (2021) · Measurement Instruments for the Social Sciences · doi:10.1186/s42409-021-00021-y · Cited by 49
Kam, C. C. S. (2016) · Sociological Methods & Research · doi:10.1177/0049124115626177 · Cited by 41
“583 2012; Quilty et al. 2006), possibly because negatively worded items can cause interpretational difficulty (Swain, Weathers, and Niedrich 2008; van Sonderen, Sanderman, and Coyne 2013).”
BackgroundJomori, M. M., Proença, R. P. D. C., Echevarría-Guanilo, M. E., Bernardo, G. L., Uggioni, P. L., & Fernandes, A. C. (2017) · British Food Journal · doi:10.1108/bfj-10-2016-0448 · Cited by 45
“Reversed worded items do not always prevent response bias and it is suggested that scores may be contaminated by respondents due to their lack of attention or confusion (van Sonderen et al., 2013).”
BackgroundJia, R., Steelman, Z. R., & Jia, H. H. (2019) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-019-03880-x · Cited by 58
Speaking the unspoken in learning analytics: troubling the defaults
Archer, E., & Prinsloo, P. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2019.1694863 · Cited by 46
Measuring Online Wellbeing: A Scoping Review of Subjective Wellbeing Measures
Ong, Z. X., Dowthwaite, L., Vallejos, E. P., Rawsthorne, M., & Long, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.616637 · Cited by 53
“This has been demonstrated in (Sonderen et al., 2013), which found that reversing the words and hence scoring led to increased respondent inattention and confusion.”
BackgroundA Short Version of the Big Five Inventory (BFI-20): Evidence on Construct Validity
Gouveia, V. V., Araújo, R. D. C. R., de Oliveira, I. C. V., Gonçalves, M. P., Milfont, T. L., Coelho, G. L. D. H., dos Santos›, W. S., de Medeiros, É. D., Soares, A. K. S., Monteiro, R. P., et al. (2021) · Revista Interamericana de Psicología/Interamerican Journal of Psychology · doi:10.30849/ripijp.v55i1.1312 · Cited by 54
“For the current version, besides showing adequate psychometric parameters (evidences on factorial validity and reliability), its five subscales or factors were composed only by positive items, which can induce response bias (van Sonderen et al., 2013).”
Career goal revision in response to negative feedback: Testing a longitudinal cross-lagged model.
Hu, S. M., Creed, P. A., & Hood, M. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000193 · Cited by 47
PsyCap and Performance: Wellbeing at Work as a Mediator
Imran, M., & Shahnawaz, M. G. (2020) · Asia-Pacific Journal of Management Research and Innovation · doi:10.1177/2319510x20915999 · Cited by 33
A Comparison of a Preliminary Version of the EQ-HWB Short and the 5-Level Version EQ-5D
Monteiro, A. L., Kuharić, M., & Pickard, A. S. (2022) · Value in Health · doi:10.1016/j.jval.2022.01.003 · Cited by 38
İlhan, M., Güler, N., Teker, G. T., & Ergenekon, Ö. (2024) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.1345549 · Cited by 52
The Australian Sheep-Goat Scale: An Evaluation of Factor Structure and Convergent Validity
Drinkwater, K. G., Denovan, A., Dagnall, N., & Parker, A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01594 · Cited by 42
Digital Storytelling Project as a Way to Engage Students in Twenty-First Century Skills Learning
Lazareva, A., & Cruz‐Martínez, G. (2020) · International Studies Perspectives · doi:10.1093/isp/ekaa017 · Cited by 52
Suh, J. K., Hwang, J., Park, S., & Hand, B. M. (2022) · Journal of Research in Science Teaching · doi:10.1002/tea.21769 · Cited by 50
Workplace Social Capital: Redefining and Measuring the Construct
Tsounis, A., Xanthopoulou, D., Demerouti, E., Kafetsios, K. G., & Tsaousis, I. (2022) · Social Indicators Research · doi:10.1007/s11205-022-03028-y · Cited by 53
Tuliao, A. P., Klanecky, A. K., Mamauag, B. L., & McChargue, D. E. (2019) · Assessment · doi:10.1177/1073191118824030 · Cited by 34
Negatively-Worded Multiple Choice Questions: An Avoidable Threat to Validity
Chiavaroli, N. G. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/ca7y-mm27 · Cited by 32
Metadata
- Authors
- Sonderen, E. V., Sanderman, R., & Coyne, J. C.
- Affiliations
- University Medical Center Groningen (NL)
- Corresponding author
- Sonderen, Eric van
- ORCID
- Sanderman, Robbert
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-31
- Volume
- 8
- Issue
- 7
- Pages
- e68967
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3729568
- OpenAlex
- W2141632279
- Cited by
- 544
- References
- 29
- Citations by year
- 2026: 44 · 2025: 50 · 2024: 49 · 2023: 48 · 2022: 54 · 2021: 59 · 2020: 70 · 2019: 51 · 2018: 44 · 2017: 27 · 2016: 29 · 2015: 16 · 2014: 3
- FWCI
- 49.55
- Type
- journal article
- Language
- en
- Update notices
- correction (2013-09-13)
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Universiteit Maastricht
- Maastricht Universitair Medisch Centrum
- Universitair Medisch Centrum Groningen
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Health Sciences › Nursing › Research and Theory › Health Education and Validation
- Keywords
- response bias, acquiescence bias, questionnaire design, Multidimensional Fatigue Inventory, fatigue measurement, psychometric properties
- MeSH terms
- Animals, Cattle, Fatigue, Humans, Psychometrics, Surveys and Questionnaires, Reproducibility of Results, Inflammatory Bowel Diseases, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.