Methods for Testing and Evaluating Survey Questionnaires

doi:10.1002/0471654728

Cited by 855Survey Methodology and NonresponseSurvey Sampling and Estimation TechniquesPsychometric Methodologies and Testing

Methods for Testing and Evaluating Survey Questionnaires

·2004·Wiley eBooks

DOI 10.1002/0471654728

Open this paper in CoChat — free account

Abstract

Contributors. Preface. 1. Methods for Testing and Evaluating Survey Questions (Stanley Presser, et al.). PART I: COGNITIVE INTERVIEWS. 2. Cognitive Interviewing Revisited: A Useful Technique, in Theory (Gordon B. Willis). 3. The Dynamics of Cognitive Interviewing (Paul Beatty). 4. Data Quality in Cognitive Interviews: The Case of Verbal Reports (Fred G. Conrad and Johnny Blair). 5. Do Different Cognitive Interview Techniques Produce Different Results (Theresa J. DeMaio and Ashley Landreth)? PART II: SUPPLEMENTS TO CONVENTIONAL PRETESTS. 6. Evaluating Survey Questions by Analyzing Patterns of Behavior Codes and Question-Answer Sequences (Johannes van der Zouwen and Johannes H. Smit). 7. Response Latency and (Para) Linguistic Expressions as Indicators of Response Error (Stasja Draisma and Wil Dijkstra). 8. Vignettes and Respondent Debriefing for Questionnaire Design and Evaluation (Elizabeth Martin). PART III: EXPERIMENTS. 9. The Case for More Split-Sample Experiments in Developing Survey Instruments (Floyd Jackson Fowler). 10. Using Field Experiments to Improve Instrument Design: The SIPP Methods Panel Project (Jeffrey Moore, et al.). 11. Experimental Design Considerations for Testing and Evaluating Questionnaires (Roger Tourangeau). PART IV: STATISTICAL MODELING. 12. Modeling Measurement Error to Identify Flawed Questions (Paul Biemer). 13. Item Response Theory (IRT) Modeling for Questionnaire Evaluation (Bryce B. Reeve and Louise C. Masse). 14. Development and Improvement of Questionnaires Using Predictions of Reliability and Validity (William E. Saris, et al.). PART V: MODE OF ADMINISTRATION. 15. Testing Paper Self-administered Questionnaires: Cognitive Interview and Field Test Comparisons (Don A. Dillman and Cleo D. Redline). 16. Methods for Testing and Evaluating Computer-Assisted Questionnaires (John Tarnai and Danna L. Moore). 17. Usability Testing to Evaluate Computer-Assisted Survey Instruments (Sue Ellen Hansen and Mick P. Couper). 18. Development and Testing of Web Questionnaires (Reginald P. Baker, et al.). PART VI: SPECIAL POPULATIONS. 19. Evolution and Adaptation of Questionnaire Development, Evaluation and Testing for Establishment Surveys (Diane K. Willimack, et al.). 20. Pretesting Questionnaires for Children and Adolescents (Edith de Leeuw, et al.). 21. Developing and Evaluating Cross-National Survey Insteuments (Tom W. Smith). 22. Survey Questionnaire Translation and Assessment (Janet Harkness, et al.). PART VII: MULTI-METHOD APPLICATIONS. 23. A Multiple-Method Approach to Improving the Clarity of Closely Related Concepts: Distinguishing Legal and Physical Custody of Children (Nora Cate Schaeffer and Jennifer Dykema). 24. Multiple Methods for Developing and Evaluating a Stated-Choice Questionnaire to Value Wetlands (Michael D. Kaplowitz, et al.). 25. Does Pretesting Make a Difference? An Experimental Test (Barbara Forsyth, et al.). References. Index.

Integrity signals

Registry

Crossref · 10.1002/0471654728

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Producing personal narratives can increase self-awareness, which can promote active coping.
“Producing personal narratives can increase self-awareness, which can, in turn, promote active coping.56,57”
From Diagnosis Blog: Checking Up on Health Blogs in the Blogosphere (2010) · cites this paper as [57]

Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · see citing works
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734 · see citing works
Untitled work
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539 · see citing works
Thinking About Answers: The Application of Cognitive Processes to Survey Methodology
Sudman, S., Bradburn, N. M., & Schwarz, N. (1995) · Cited by 1,586 · see citing works
Survey Methods in Social Investigation.
T., C. G., & Moser, C. A. (1958) · The Incorporated Statistician · doi:10.2307/2986383 · Cited by 1,438 · see citing works
Measurement and prediction
Stouffer, S. A. (1950) · Princeton University Press eBooks · Cited by 1,244 · see citing works
Asking questions: A practical guide to questionnaire design
(1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90052-u · Cited by 868 · see citing works
Thinking about Answers: The Application of Cognitive Process to Survey Methodology
Aalbaum, G., Sudman, S., Bradburn, N. M., & Schwarz, N. (1996) · Journal of Marketing Research · doi:10.2307/3152133 · Cited by 783 · see citing works
Survey Questions: Handcrafting the Standardized Questionnaire
Quarm, D., Converse, J. M., & Presser, S. (1988) · Teaching Sociology · doi:10.2307/1317544 · Cited by 767 · see citing works
The Science of Asking Questions
Schaeffer, N. C., & Presser, S. (2003) · Annual Review of Sociology · doi:10.1146/annurev.soc.29.110702.110112 · Cited by 625 · see citing works
Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines.
Schaible, W. L., Jabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (1987) · Journal of the American Statistical Association · doi:10.2307/2289187 · Cited by 618 · see citing works
Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach
Andrews, F. M. (1984) · Public Opinion Quarterly · doi:10.1086/268840 · Cited by 419 · see citing works
The Pretest in Survey Research: Issues and Preliminary Findings
Hunt, S. D., Sparkman, R. D., & Wilcox, J. B. (1982) · Journal of Marketing Research · doi:10.2307/3151627 · Cited by 373 · see citing works
Survey questionnaire translation and assessment
Harkness, J., Pennell, B., & Schoua‐Glusberg, A. (2004) · Methods for Testing and Evaluating Survey Questionnaires · doi:10.1002/0471654728.ch22 · Cited by 370 · see citing works
Survey Pretesting: Do Different Methods Produce Different Results?
Presser, S., & Blair, J. (1994) · Sociological Methodology · doi:10.2307/270979 · Cited by 275 · see citing works
Questionnaire Construction and Item Writing
Sheatsley, P. B. (1983) · Elsevier eBooks · doi:10.1016/b978-0-12-598226-9.50012-4 · Cited by 251 · see citing works
The Design and Understanding of Survey Questions.
Morton-Williams, J., & Belson, W. A. (1984) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981742 · Cited by 250 · see citing works
Computer Assisted Survey Information Collection
(2000) · Technometrics · doi:10.1080/00401706.2000.10486039 · Cited by 202 · see citing works
The Meaning of Questions
Hudson, R. (1975) · Language · doi:10.2307/413148 · Cited by 161 · see citing works
Response Latency as a Signal to Question Problems in Survey Research
Bassili, J. N., & Scott, B. S. (1996) · Public Opinion Quarterly · doi:10.1086/297760 · Cited by 136 · see citing works
Pretesting Questionnaires for Children and Adolescents
de Leeuw, E. D., Borgers, N., & Smits, A. (2004) · doi:10.1002/0471654728.ch20 · Cited by 130 · see citing works
Cognitive Interviewing Revisited: A Useful Technique, in Theory?
Willis, G. (2004) · doi:10.1002/0471654728.ch2 · Cited by 103 · see citing works
Developing and Evaluating Cross‐National Survey Instruments
Smith, T. (2004) · doi:10.1002/0471654728.ch21 · Cited by 95 · see citing works
The Dynamics of Cognitive Interviewing
Beatty, P. C. W. (2004) · doi:10.1002/0471654728.ch3 · Cited by 89 · see citing works
A coding system for appraising questionnaires.
Lessler, J. T., & Forsyth, B. (1996) · Cited by 64 · see citing works
Do Different Cognitive Interview Techniques Produce Different Results?
DeMaio, T. J., & Landreth, A. (2004) · doi:10.1002/0471654728.ch5 · Cited by 63 · see citing works
Pretesting of Questionnaires
Sletto, R. F. (1940) · American Sociological Review · doi:10.2307/2083634 · Cited by 63 · see citing works
Response Latency and (Para)Linguistic Expressions as Indicators of Response Error
Draisma, S., & Dijkstra, W. (2004) · doi:10.1002/0471654728.ch7 · Cited by 61 · see citing works
Multiple Methods for Developing and Evaluating a Stated‐Choice Questionnaire to Value Wetlands
Kaplowitz, M. D., Lupi, F., & Hoehn, J. P. (2004) · doi:10.1002/0471654728.ch24 · Cited by 43 · see citing works
Vignettes and Respondent Debriefing for Questionnaire Design and Evaluation
Martin, E. (2004) · doi:10.1002/0471654728.ch8 · Cited by 39 · see citing works
The Case for More Split‐Sample Experiments in Developing Survey Instruments
Fowler, F. J. (2004) · doi:10.1002/0471654728.ch9 · Cited by 38 · see citing works
Questionnaire pretesting: Computer-assisted coding of concurrent protocols.
Bolton, R. N., & Bronkhorst, T. M. (1996) · Cited by 26 · see citing works
Questionnaire design in the cognitive research laboratory
Lessler, J. T. (1989) · Cited by 25 · see citing works
Modeling Measurement Error to Identify Flawed Questions
Biemer, P. P. (2004) · doi:10.1002/0471654728.ch12 · Cited by 23 · see citing works
Evolution and Adaptation of Questionnaire Development, Evaluation, and Testing Methods for Establishment Surveys
Willimack, D. K., Lyberg, L. E., Martin, J., et al. (2004) · doi:10.1002/0471654728.ch19 · Cited by 23 · see citing works
Experimental Design Considerations for Testing and Evaluating Questionnaires
Tourangeau, R. (2004) · doi:10.1002/0471654728.ch11 · Cited by 22 · see citing works
Development and Improvement of Questionnaires Using Predictions of Reliability and Validity
Saris, W. E., van der Veld, W., & Gallhofer, I. N. (2004) · doi:10.1002/0471654728.ch14 · Cited by 20 · see citing works
Development and Testing of Web Questionnaires
Baker, R. P., Crawford, S. D., & Swinehart, J. (2004) · doi:10.1002/0471654728.ch18 · Cited by 19 · see citing works
Usability Testing to Evaluate Computer‐Assisted Instruments
Hansen, S. E., & Couper, M. P. (2004) · doi:10.1002/0471654728.ch17 · Cited by 15 · see citing works
I. THE MEANING OF QUESTIONS
Cantril, H. (1944) · Princeton University Press eBooks · doi:10.1515/9781400877546-002 · Cited by 12 · see citing works
Toward a Social Psychological Programme for Improving Focus Group Methods of Developing Questionnaires
Bischoping, K., & Dykema, J. (1999) · Journal of Official Statistics · Cited by 11 · see citing works
Three Criteria: Knowledge, Conviction, and Significance
Katz, D. M. (1940) · Public Opinion Quarterly · doi:10.1086/265403 · Cited by 11 · see citing works
Methods for Testing and Evaluating Computer‐Assisted Questionnaires
Tarnai, J., & Moore, D. L. (2004) · doi:10.1002/0471654728.ch16 · Cited by 8 · see citing works
The Collection of Data
Moore, P. G. (1969) · Cambridge University Press eBooks · doi:10.1017/cbo9780511569685.003 · Cited by 5 · see citing works

Flagged first, then by citations

Sorted by citations

Research synthesis: the practice of cognitive interviewing
Beatty, P. C., & Willis, G. B. (2007) · Public Opinion Quarterly · doi:10.1093/poq/nfm006 · Cited by 1,615 · see citing works
Content Validity—Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2—Assessing Respondent Understanding
Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., & Ring, L. (2011) · Value in Health · doi:10.1016/j.jval.2011.06.013 · Cited by 1,135 · see citing works
Ten Steps in Scale Development and Reporting: A Guide for Researchers
Carpenter, S. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2017.1396583 · Cited by 1,141 · see citing works
The prevalence of child sexual abuse in community and student samples: a meta-analysis
Pereda, N., Guilera, G., Forns, M., & Gómez-Benito, J. (2009) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.02.007 · Cited by 1,068 · see citing works
What Is Sufficient Evidence for the Reliability and Validity of Patient-Reported Outcome Measures?
Frost, M. H., Reeve, B. B., Liepa, A. M., Stauffer, J. W., & Hays, R. D. (2007) · Value in Health · doi:10.1111/j.1524-4733.2007.00272.x · Cited by 572 · see citing works
PRO development: rigorous qualitative research as the crucial foundation
Lasch, K. E., Marquis, P., Vigneux, M., Abetz, L., Arnould, B., Bayliss, M., Crawford, B., & Rosa, K. (2010) · Quality of Life Research · doi:10.1007/s11136-010-9677-6 · Cited by 461 · see citing works
Stuck in the middle: The use and interpretation of mid-points in items on questionnaires
Nadler, J. T., Weston, R., & Voyles, E. C. (2015) · The Journal of General Psychology · doi:10.1080/00221309.2014.994590 · Cited by 396 · see citing works
Interviewing to develop Patient-Reported Outcome (PRO) measures for clinical research: eliciting patients’ experience
Brédart, A., Marrel, A., Abetz-Webb, L., Lasch, K., & Acquadro, C. (2014) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-12-15 · Cited by 195 · see citing works
Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207 · see citing works
Defining clinical severity in adults with obsessive–compulsive disorder
Storch, E. A., De Nadai, A. S., Conceição do Rosário, M., Shavitt, R. G., Torres, A. R., Ferrão, Y. A., Miguel, E. C., Lewin, A. B., & Fontenelle, L. F. (2015) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2015.08.007 · Cited by 133 · see citing works
Diagnosis Blog: Checking Up on Health Blogs in the Blogosphere
Miller, E. A., & Pole, A. (2010) · American Journal of Public Health · doi:10.2105/ajph.2009.175125 · Cited by 129 · see citing works
Effects of gamification on participation and data quality in a real-world market research domain
Cechanowicz, J., Gutwin, C., Brownell, B., & Goodfellow, L. (2013) · Proceedings of the First International Conference on Gameful Design, Research, and Applications · doi:10.1145/2583008.2583016 · Cited by 103 · see citing works
Challenging the Construct Validity of Mindfulness Assessment—a Cognitive Interview Study of the Freiburg Mindfulness Inventory
Belzer, F., Schmidt, S., Lucius-Hoene, G., Schneider, J. F., Orellana-Rios, C. L., & Sauer, S. (2013) · Mindfulness · doi:10.1007/s12671-012-0165-7 · Cited by 88 · see citing works
Comparing results from different styles of cognitive interviewing: ‘verbal probing’ vs. ‘thinking aloud’
Priede, C., & Farrall, S. (2011) · International Journal of Social Research Methodology · doi:10.1080/13645579.2010.523187 · Cited by 88 · see citing works
Hypothetical Vignettes in Empirical Bioethics Research
Ulrich, C. M., & Ratcliffe, S. J. (2007) · Advances in Bioethics · doi:10.1016/s1479-3709(07)11008-6 · Cited by 56 · see citing works
Examining how rural ecological contexts influence children’s early learning opportunities
Iruka, I. U., DeKraai, M., Walther, J., Sheridan, S. M., & Abdel-Monem, T. (2020) · Early Childhood Research Quarterly · doi:10.1016/j.ecresq.2019.09.005 · Cited by 51 · see citing works
Optimizing household survey methods to monitor the Sustainable Development Goals targets 6.1 and 6.2 on drinking water, sanitation and hygiene: A mixed-methods field-test in Belize
Khan, S. M., Bain, R. E. S., Lunze, K., Unalan, T., Beshanski-Pedersen, B., Slaymaker, T., Johnston, R., & Hancioglu, A. (2017) · PLOS ONE · doi:10.1371/journal.pone.0189089 · Cited by 46 · see citing works
Survey Design for Visually Impaired and Blind People
Kaczmirek, L., & Wolff, K. G. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-73279-2_41 · Cited by 38 · see citing works
Psychometric properties of the Perceived Stress Scale in a sample of German dementia patients and their caregivers
Deeken, F., Häusler, A., Nordheim, J., Rapp, M., Knoll, N., & Rieckmann, N. (2018) · International Psychogeriatrics · doi:10.1017/s1041610217001387 · Cited by 30 · see citing works
Self-care adherence and psychological functioning of older patients with type 2 diabetes: effects of persuasion, social pressure, and self-efficacy
Yang, F., Pang, J. S., & Cheng, W. J. Y. (2016) · Journal of Clinical Psychology in Medical Settings · doi:10.1007/s10880-016-9470-y · Cited by 12 · see citing works

Venue: Wiley eBooks
Publisher: Wiley
Published: 2004-06-25
DOI: 10.1002/0471654728
OpenAlex: W2119046183
Cited by: 855
References: 44
Citations by year: 2026: 16 · 2025: 14 · 2024: 17 · 2023: 12 · 2022: 12 · 2021: 46 · 2020: 29 · 2019: 47 · 2018: 45 · 2017: 44 · 2016: 60 · 2015: 55 · 2014: 79 · 2013: 55 · 2012: 60
FWCI: 45.28
Type: book
Language: en
Open access: closed
Licence: doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
Locations: Wiley eBooks (publishedVersion)
Topics: Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Keywords: questionnaire evaluation, cognitive interviewing, pretest, response latency, response errors, vignettes, field experiments, measurement error modeling, item response theory, reliability and validity, online surveys, establishment surveys, usability testing, verbal protocols

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Methods for Testing and Evaluating Survey Questionnaires

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks