The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
An Examination of Questionnaire Evaluation by Expert Reviewers
Abstract
Expert reviews are frequently used as a questionnaire evaluation method but have received little empirical attention. Questions from two surveys are evaluated by six expert reviewers using a standardized evaluation form. Each of the questions has validation data available from records. Large inconsistencies in ratings across the six experts are found. Despite the lack of reliability, the average expert ratings successfully identify questions that had higher item nonresponse rates and higher levels of inaccurate reporting. This article provides empirical evidence that experts are able to discern questions that manifest data quality problems, even if individual experts vary in what they rate as being problematic. Compared to a publicly available computerized question evaluation tool, ratings by the human experts positively predict questions with data quality problems, whereas the computerized tool varies in success in identifying these questions. These results indicate that expert reviews have value in identifying question problems that result in lower survey data quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Stefanski, L. A., Fuller, W., & Wiley, J. (1988) · Journal of Business & Economic Statistics · doi:10.2307/1391893 · Cited by 2,794
Fuller, W. A. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316665 · Cited by 2,351
Beyond kappa: a review of interrater agreement measures
Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999) · Canadian Journal of Statistics · doi:10.2307/3315487 · Cited by 976
AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH
STAR, S. A. (1980) · Public Opinion Quarterly · doi:10.1093/poq/44.3.420 · Cited by 814
Research on Interviewing Techniques
Cannell, C. F., Miller, P. V., & Oksenberg, L. (1981) · Sociological Methodology · doi:10.2307/270748 · Cited by 436
Survey Pretesting: Do Different Methods Produce Different Results?
Presser, S., & Blair, J. (1994) · Sociological Methodology · doi:10.2307/270979 · Cited by 274
Intraclass correlations: There's more there than meets the eye.
Lahey, M. A., Downey, R. G., & Saal, F. E. (1983) · Psychological Bulletin · doi:10.1037//0033-2909.93.3.586 · Cited by 222
Clarifying Question Meaning in a Household Telephone Survey
Conrad, F. G., & Schober, M. F. (2000) · Public Opinion Quarterly · doi:10.1086/316757 · Cited by 196
Cited by
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
How and why are educators using Instagram?
Carpenter, J. P., Morrison, S. A., Craft, M., & Lee, M. (2020) · Teaching and Teacher Education · doi:10.1016/j.tate.2020.103149 · Cited by 277
Squires, A., Aiken, L. H., van den Heede, K., Sermeus, W., Bruyneel, L., Lindqvist, R., Schoonhoven, L., Stromseng, I., Busse, R., Brzostek, T., et al. (2013) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2012.02.015 · Cited by 265
Ikart, E. M. (2019) · Asian Journal of Social Science Studies · doi:10.20849/ajsss.v4i2.565 · Cited by 140
Development and validation of the Food Noise Questionnaire
Diktas, H. E., Cardel, M. I., Foster, G. D., LeBlanc, M. M., Dickinson, S. L., Ables, E. M., Chen, X., Nathan, R., Shapiro, D., & Martin, C. K. (2025) · Obesity · doi:10.1002/oby.24216 · Cited by 35
Metadata
- Authors
- Olson, K.
- Venue
- Field Methods
- Publisher
- SAGE Publications
- Published
- 2010-08-20
- Volume
- 22
- Issue
- 4
- Pages
- 295-318
- ISSN
- 1064-8631 (ISSN-L)
- 1525-822X
- 1552-3969
- OpenAlex
- W2161421195
- Cited by
- 253
- References
- 21
- FWCI
- 5.22
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Examination of Questionnaire Evaluation by Expert Reviewers. https://app.cochat.ai/papers/10.1177/1525822x10379795
- The paper
- Olson, K. (2010). An Examination of Questionnaire Evaluation by Expert Reviewers. Field Methods. https://doi.org/10.1177/1525822x10379795
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.