It’s Not Me, It’s You: Miscomprehension in Surveys
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The ubiquity of surveys in organizational research means that their quality is of paramount importance. Commonly this has been addressed through the use of sophisticated statistical approaches with scant attention paid to item comprehension. Linguistic theory suggests that while everyone may understand an item, they may comprehend it in different ways. We explore this in two studies in which we administered three published scales and asked respondents to indicate what they believed the items meant, and a third study that replicated the results with an additional scale. These demonstrate three forms of miscomprehension: instructional (where instructions are not followed), sentential (where the syntax of a sentence is enriched or depleted as it is interpreted), and lexical (where different meanings of words are deployed). These differences in comprehension are not appreciable using conventional statistical analyses yet can produce significantly different results and cause respondents to tap into different concepts. These results suggest that item interpretation is a significant source of error, which has been hitherto neglected in the organizational literature. We suggest remedies and directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539
A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires
Hinkin, T. R. (1998) · Organizational Research Methods · doi:10.1177/109442819800100106 · Cited by 4,095
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · cited 1×
“This is the consistent with the methods described by Meade and Craig (2012) to detect and eliminate cases in which the respondent is not attending to the content at all.”
MethodInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
A Review of Scale Development Practices in the Study of Organizations
Hinkin, T. R. (1995) · Journal of Management · doi:10.1177/014920639502100509 · Cited by 3,193 · cited 4×
“Despite the importance of the initial stages of item development (Hinkin, 1995), greater emphasis is often placed on the statistical assessment of the Hardy and Ford 139”
BackgroundInfluentialConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 2×
“The survey was administered using Amazon’s Mechanical Turk (mTurk) to obtain a sample of respondents who were currently employed. mTurk has been used effectively in various fields, including both linguistic and psychology studies (see Mason & Suri, 2012; Sprouse, 2011).”
MethodResultSelf-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846 · cited 2×
“The importance of linguistics in item interpretation has not been widely discussed although there are some existing sources that do address the issue (Schwarz, 1999), and it is to this topic we now turn.”
Brayfield, A. H., & Rothe, H. F. (1951) · Journal of Applied Psychology · doi:10.1037/h0055617 · Cited by 2,565
McFarlin, D. B., & Sweeney, P. D. (1992) · Academy of Management Journal · doi:10.2307/256489 · Cited by 1,704 · cited 1×
“Two multiple item scales for job satisfaction (Agho et al., 1992, 6 items; Tsui et al., 1992, 6 items) and one for procedural justice (McFarlin & Sweeney, 1992, 4 items) were used.”
Research synthesis: the practice of cognitive interviewing
Beatty, P. C., & Willis, G. B. (2007) · Public Opinion Quarterly · doi:10.1093/poq/nfm006 · Cited by 1,615 · cited 1×
“In public opinion surveys cognitive interviewing is commonly used to pretest items (Beatty & Willis, 2007; Schwarz & Sudman, 1996).”
BackgroundNull hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090 · cited 1×
“This suggests that sentential miscomprehension can have indirect effects, as such a result is unlikely to have happened by chance (Nickerson, 2000).”
BackgroundAnderson, J. C., & Gerbing, D. W. (1991) · Journal of Applied Psychology · doi:10.1037/0021-9010.76.5.732 · Cited by 1,038
Foundations of Multimethod Research
Brewer, J., & Hunter, A. (2006) · doi:10.4135/9781412984294 · Cited by 287
Cited by
A Review of Key Likert Scale Development Advances: 1995–2019
Jebb, A. T., Ng, V., & Tay, L. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.637547 · Cited by 770
Simmering, M. J., Fuller, C. M., Richardson, H. A., Ocal, Y., & Atinc, G. M. (2015) · Organizational Research Methods · doi:10.1177/1094428114560023 · Cited by 734
Schmiedel, T., Müller, O., & vom Brocke, J. (2019) · Organizational Research Methods · doi:10.1177/1094428118773858 · Cited by 319
“For example, the generalizability of case study data and the appropriateness of using questionnaires to gain valid insights are often discussed issues (Hardy & Ford, 2014; Schein, 1990).”
BackgroundKeith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
A critique of the Leader-Member Exchange construct: Back to square one
Gottfredson, R. K., Wright, S. L., & Heaphy, E. D. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101385 · Cited by 205
Metadata
- Authors
- Hardy, B., & Ford, L. R.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2014-02-07
- Volume
- 17
- Issue
- 2
- Pages
- 138-162
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2155377391
- Cited by
- 69
- References
- 15
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.