Research synthesis: the practice of cognitive interviewing
Abstract
Cognitive interviewing has emerged as one of the more prominent methods for identifying and correcting problems with survey questions. We define cognitive interviewing as the administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends. But beyond this general categorization, cognitive interviewing potentially includes a variety of activities that may be based on different assumptions about the type of data that are being collected and the role of the interviewer in that process. This synthesis reviews the range of current cognitive interviewing practices, focusing on three considerations: (1) what are the dominant paradigms of cognitive interviewing—what is produced under each, and what are their apparent advantages; (2) what key decisions about cognitive interview study design need to be made once the general approach is selected (e.g., who should be interviewed, how many interviews should be conducted, and how should probes be selected), and what bases exist for making these decisions; and (3) how cognitive interviewing data should be evaluated, and what standards of evidence exist for making questionnaire design decisions based on study findings. In considering these issues, we highlight where standards for best practices are not clearly defined, and suggest broad areas worthy of additional methodological research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cognitive interviewing was used to conduct a pre-test that guided the adaptation process.
“Cognitive interviewing was used to conduct a pre-test that guided the adaptation process.”
From Translation, adaptation, and initial evaluation of a guided self-help intervention to reduce psychological distress among nurses during COVID-19 in China (2023) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Basics of Qualitative Research: Grounded Theory Procedures and Techniques.
Ellis, C., Strauss, A., & Corbin, J. M. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074814 · Cited by 29,279 · see citing works
Basics of Qualitative Research: Grounded Theory Procedures and Techniques
Long, D. R., Strauss, A., & Corbin, J. M. (1993) · Modern Language Journal · doi:10.2307/328955 · Cited by 17,612 · see citing works
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317 · see citing works
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539 · see citing works
Book Review: Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,361 · see citing works
Holstein, J. A., & Gubrium, J. F. (1995) · Sage eBooks · doi:10.4135/9781412986120 · Cited by 3,074 · see citing works
Cognitive Interviewing: A Tool for Improving Questionnaire Design
Willis, G. (2004) · Cited by 1,934 · see citing works
Self-citation(1992) · Cited by 1,155 · see citing works
The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes
van Someren, M. (1994) · Cited by 975 · see citing works
Methods for Testing and Evaluating Survey Questionnaires
(2004) · doi:10.1002/0471654728 · Cited by 855 · see citing works
Survey Questions: Handcrafting the Standardized Questionnaire
Quarm, D., Converse, J. M., & Presser, S. (1988) · Teaching Sociology · doi:10.2307/1317544 · Cited by 767 · see citing works
The think aloud method: A practical guide to modelling cognitive processes
Solomon, P. J. (1995) · Information Processing & Management · doi:10.1016/0306-4573(95)90031-4 · Cited by 748 · see citing works
Stewart, D. W., Schwarz, N., & Sudman, S. (1996) · Journal of Marketing Research · doi:10.2307/3152152 · Cited by 694 · see citing works
Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines.
Schaible, W. L., Jabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (1987) · Journal of the American Statistical Association · doi:10.2307/2289187 · Cited by 618 · see citing works
The validity of verbal protocols
Russo, J. E., Johnson, E. J., & STEPHENS, D. L. (1989) · Memory & Cognition · doi:10.3758/bf03202637 · Cited by 545 · see citing works
Standardized Survey Interviewing: Minimizing Interviewer Related Error.
Sudman, S., Fowler, F. J., & Mangione, T. W. (1992) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075522 · Cited by 367 · see citing works
The use of verbal report methods in the development and testing of survey questionnaires
Willis, G., Royston, P., & Bercini, D. (1991) · Applied Cognitive Psychology · doi:10.1002/acp.2350050307 · Cited by 290 · see citing works
Self-citationSurvey Pretesting: Do Different Methods Produce Different Results?
Presser, S., & Blair, J. (1994) · Sociological Methodology · doi:10.2307/270979 · Cited by 275 · see citing works
Cognition, Aging and Self-Reports
(1998) · Psychology Press eBooks · doi:10.4324/9780203345115 · Cited by 257 · see citing works
The Design and Understanding of Survey Questions.
Morton-Williams, J., & Belson, W. A. (1984) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981742 · Cited by 250 · see citing works
Cognitive Laboratory Methods: A Taxonomy
Forsyth, B., & Lessler, J. T. (2004) · Wiley series in probability and statistics · doi:10.1002/9781118150382.ch20 · Cited by 154 · see citing works
Cognitive interviewing techniques: In the lab and in the field.
DeMaio, T. J., & Rothgeb, J. M. (1996) · Cited by 139 · see citing works
Ziegel, E. R., Sirken, M. G., Herrmann, D. J., Schechter, S., Schwarz, N., Tanur, J. M., & Tourangeau, R. (2000) · Technometrics · doi:10.2307/1271121 · Cited by 113 · see citing works
Methods for Testing and Evaluating Survey Questions
Presser, S., Couper, M. P., Lessler, J. T., et al. (2004) · doi:10.1002/0471654728.ch1 · Cited by 109 · see citing works
Standardized Survey Interviewing: Minimizing Interviewer-Related Error.
O'Rourke, D. M., Fowler, F. J., & Mangione, T. W. (1991) · Journal of the American Statistical Association · doi:10.2307/2289751 · Cited by 106 · see citing works
Cognitive Interviewing Revisited: A Useful Technique, in Theory?
Willis, G. (2004) · doi:10.1002/0471654728.ch2 · Cited by 103 · see citing works
Self-citationThe Dynamics of Cognitive Interviewing
Beatty, P. C. W. (2004) · doi:10.1002/0471654728.ch3 · Cited by 89 · see citing works
Self-citationData Quality in Cognitive Interviews: The Case of Verbal Reports
Conrad, F. G., & Blair, J. (2004) · doi:10.1002/0471654728.ch4 · Cited by 66 · see citing works
Evaluation of Cognitive Interviewing Techniques: Do the Results Generalize to the Field?
Willis, G., & Schechter, S. (1997) · Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique · doi:10.1177/075910639705500105 · Cited by 50 · see citing works
Self-citationAn Empirical Evaluation of In-Depth Probes Used to Pretest Survey Questions
Foddy, W. (1998) · Sociological Methods & Research · doi:10.1177/0049124198027001003 · Cited by 45 · see citing works
Hak, T., van der Veer, C. G., & Jansen, H. A. M. (2004) · RePub (Erasmus University Rotterdam) · Cited by 33 · see citing works
Perspectives on Pretesting : "Cognition" in the Cognitive Interview?
Gerber, E. R., & Wellens, T. R. (1997) · Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique · doi:10.1177/075910639705500104 · Cited by 30 · see citing works
Questionnaire pretesting: Computer-assisted coding of concurrent protocols.
Bolton, R. N., & Bronkhorst, T. M. (1996) · Cited by 26 · see citing works
Asking Survey Respondents About Health Status
Schechter, S., Beatty, P. C. W., & Willis, G. (2010) · Taylor & Francis eBooks · doi:10.4324/9780203345115_chapter_13 · Cited by 22 · see citing works
Self-citationQUESTIONNAIRE DESIGN RESEARCH LABORATORY
Royston, P., Bercini, D., Sirken, M. G., et al. (1985) · Cited by 20 · see citing works
Pretesting questionnaires in the laboratory: an alternative approach.
Bercini, D. (1992) · PubMed · Cited by 19 · see citing works
Methodological Issues Surrounding the Application of Cognitive Psychology in Survey Research
Tucker, C. (1997) · Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique · doi:10.1177/075910639705500106 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 29
- Bates, Nancy, and Theresa J. DeMaio. 1989. "Using Cognitive Research Methods to Improve the Design of the Decennial Census Form." Proceedings of the U.S. Bureau of the Census Annual Research Conference 267-77.
- Beatty, Paul. 2002. "Cognitive Interview Evaluation of the Blood Donor History Screening Questionnaire." In Final Report of the AABB Task Force to Redesign the Blood Donor Screening Questionnaire. Report submitted to the U.S. Food and Drug Administration. ______. 2003. Answerable Questions: Advances in the Methodology for Identifying and Resolving Questionnaire Problems in Survey Research. Doctoral Dissertation, the University of Michigan In Dissertation Abstracts International 64(09): 3504A.
- Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith T. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. Hoboken, NJ: John Wiley and Sons.
- Beatty, Paul, Floyd J. Fowler, and Carol Cosenza. 2006. "Do Questionnaire Design Recommendations Lead to Measurable Improvements? Some Experiments with Alternate Versions of Complex Survey Questions." Proceedings of the Q2006 European Conference on Quality in Survey Statistics. Available online at: http://www.statistics.gov.uk/events/q2006/ downloads/T04_Beatty.doc.
- Blair, Johnny, Frederick Conrad, Allison Ackermann, and Greg Claxton. 2006. "The Effect of Sample Size on Cognitive Interview Findings." Paper presented at the American Association for Public Opinion Research Conference, Montreal, Quebec, Canada.
- Conrad, Frederick, and Johnny Blair. 1996. "From Impressions to Data: Increasing the Objectivity of Cognitive Interviews." 1996 Proceedings of the Section on Survey Research Methods, Vol. 1, pp. 1-9. Alexandria, VA: American Statistical Association.
- Conrad, Frederick, Johnny Blair, and Elena Tracy. 2000. "Verbal Reports Are Data! A Theoretical Approach to Cognitive Interviews." In Proceedings of the 1999 Federal Committee on Statistical Methodology Research Conference. Washington, DC: Office of Management and Budget.
- DeMaio, Theresa J., Susan Ciochetto, and Wendy Davis. 1993. "Research on the Continuing Survey of Food Intakes by Individuals." 1993 Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 1021-6. Alexandria, VA: American Statistical Association.
- Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith T. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. Hoboken, NJ: John Wiley and Sons.
- Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith T. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. Hoboken, NJ: John Wiley and Sons.
- Ericsson, K. Anders., and Herbert A. Simon. 1980. "Verbal Reports as Data." Psychological Review 87:215-51.
- ______. 1993. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: MIT Press.
- Paul P. Biemer, Robert M. Groves, Lars E. Lyberg, Nancy A. Mathiowetz, and Seymour Sudman. New York: Wiley.
- Stanley Presser, Jennifer M. Rothgeb, Mick P. Couper, Judith T. Lessler, Elizabeth Martin, Jean Martin, and Eleanor Singer. Hoboken, NJ: John Wiley and Sons.
- Monroe G. Sirken, Douglas J. Herrmann, Susan Schechter, Norbert Schwarz, Judith M. Tanur, and Roger Tourangeau. New York: Wiley.
- Thomas B. Jabine, Miron L. Straf, Judith M. Tanur, and Roger Tourangeau. Washington, DC: National Academy Press.
- Miller, Kristen. 2002. "The Role of Social Location in Question Response: Rural Poor Experience Answering General Health Questions." Paper presented at the American Association for Public Opinion Research Conference held in St. Pete Beach, Florida, May 2002.
- ______. 2005. "Q-Bank: Development of a Tested-Question Database." 2005 Proceedings of the Section on Government Statistics, pp. 1352-9. Alexandria, VA: American Statistical Association.
- O'Muircheartaigh, Colm. 1999. "CASM: Successes, Failures, and Potential." In Cognition and Survey Research, ed. Monroe G. Sirken, Douglas J. Herrmann, Susan Schechter, Norbert Schwarz, Judith M. Tanur, and Roger Tourangeau. New York: Wiley.
- Oksenberg, Lois, Charles F. Cannell, and Graham Kalton. 1991. "New Strategies for Pretesting Survey Questions." Journal of Official Statistics 7:349-65.
- Presser, Stanley. 1989. "Pretesting: A Neglected Aspect of Survey Research.'" In Conference Proceedings: Health Survey Research Methods, ed.
- Redline, Cleo, Richard Smiley, Meredith Lee, Theresa DeMaio, and Don Dillman, D. 1998. "Beyond Concurrent Interviews: An Evaluation of Cognitive Interviewing Techniques for Self- Administered Questionnaires." 1998 Proceedings of the Section on Survey Research Methods, pp. 900-5, Alexandria, VA: American Statistical Association.
- Royston, Patricia N. 1989. "Using Intensive Interviews to Evaluate Questions." In Conference Proceedings: Health Survey Research Methods, ed.
- Royston, Patricia N., and Deborah Bercini. 1987. "Questionnaire Design Research in a Laboratory Setting: Results of Testing Cancer Risk Factor Questions." 1987 Proceedings of the Section on Survey Research Methods, pp. 829-33. Alexandria, VA: American Statistical Association.
- Norbert Schwarz, Denise Park, Barbel Knauper, and Seymour Sudman. Philadelphia, PA: Psychology Press.
- Schechter, Susan, Johnny Blair, and Janet Vande Hey. 1996. "Conducting Cognitive Interviews to Test Self-Administered and Telephone Surveys: Which Methods Should We Use?" 1996 Proceedings of the Section on Survey Research Methods, pp. 10-7. Alexandria, VA: American Statistical Association.
- Monroe G. Sirken, Douglas J. Herrmann, Susan Schechter, Norbert Schwarz, Judith M. Tanur, and Roger Tourangeau. New York: Wiley.
- Weiss, Robert S. 1994. Learning from Strangers: The Art and Method of Qualitative Interviewing Studies. New York: The Free Press.
- Monroe G. Sirken, Douglas J. Herrmann, Susan Schechter, Norbert Schwarz, Judith M. Tanur, and Roger Tourangeau. New York: Wiley.
Cited by
Sorted by citations
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018) · Frontiers in Public Health · doi:10.3389/fpubh.2018.00149 · Cited by 5,925 · see citing works
What about the girls? Sex-based differences in autistic traits and adaptive skills
Ratto, A. B., Kenworthy, L., Yerys, B. E., Bascom, J., Wieckowski, A. T., White, S. W., Wallace, G. L., Pugliese, C., Schultz, R. T., Ollendick, T. H., et al. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-017-3413-9 · Cited by 342 · see citing works
Green, C. A., Duan, N., Gibbons, R. D., Hoagwood, K. E., Palinkas, L. A., & Wisdom, J. P. (2015) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-014-0552-6 · Cited by 205 · see citing works
Farvardin, S., Patel, J., Khambaty, M., Yerokun, O. A., Mok, H., Tiro, J. A., Yopp, A. C., Parikh, N. D., Marrero, J. A., & Singal, A. G. (2017) · Hepatology · doi:10.1002/hep.28770 · Cited by 185 · see citing works
Leyenaar, J. K., Freyleue, S. D., Bordogna, A., Wong, C., Penwill, N., & Bode, R. (2021) · JAMA · doi:10.1001/jama.2021.18377 · Cited by 90 · see citing works
Priede, C., & Farrall, S. (2011) · International Journal of Social Research Methodology · doi:10.1080/13645579.2010.523187 · Cited by 88 · see citing works
Tol, W. A., Augustinavicius, J., Carswell, K., Brown, F. L., Adaku, A., Leku, M. R., García-Moreno, C., Ventevogel, P., White, R. G., & van Ommeren, M. (2018) · Global Mental Health · doi:10.1017/gmh.2018.14 · Cited by 69 · see citing works
“Cognitive interviewing is commonly used to check whether the meaning of survey questions has been translated accurately (Beatty & Willis, 2007), but has also been applied in the development of interventions such as health education materials (Carbone et al., 2002).”
BackgroundDevelopment and evaluation of the Internalized Racism in Asian Americans Scale (IRAAS)
Choi, A. Y., Israel, T., & Maeda, H. (2017) · Journal of Counseling Psychology · doi:10.1037/cou0000183 · Cited by 70 · see citing works
Political engagement and wellbeing among college students
Ballard, P. J., Ni, X., & Brocato, N. (2020) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2020.101209 · Cited by 61 · see citing works
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58 · see citing works
Smith, A. “., Bamgboje‐Ayodele, A., Butow, P., Klein, B., Turner, J., Sharpe, L., Fardell, J., Beatty, L., Pearce, A., Thewes, B., et al. (2020) · Psycho-Oncology · doi:10.1002/pon.5218 · Cited by 54 · see citing works
Beidas, R. S., Maclean, J. C., Fishman, J., Dorsey, S., Schoenwald, S. K., Mandell, D. S., Shea, J. A., McLeod, B. D., French, M. T., Hogue, A., et al. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-1034-z · Cited by 50 · see citing works
Differences in Coaching Needs Among Underrepresented Minority Medical Students
Najibi, S., Carney, P. A., Thayer, E. K., & Deiorio, N. M. (2019) · Family Medicine · doi:10.22454/fammed.2019.100305 · Cited by 30 · see citing works
Daouk-Öyry, L., & McDowal, A. (2013) · Journal of Personality Assessment · doi:10.1080/00223891.2012.735300 · Cited by 25 · see citing works
Pacichana-Quinayáz, S. G., Osorio-Cuéllar, G. V., Bonilla-Escobar, F. J., Fandiño-Losada, A., & Gutiérrez-Martínez, M. I. (2016) · Ciência & Saúde Coletiva · doi:10.1590/1413-81232015216.07062015 · Cited by 25 · see citing works
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20 · see citing works
“…questions about experience and attitudes towards AI and were tested cognitively following the “think aloud” paradigm identified by Beatty and Willis [43]: we tested participants’ understanding of the questions, ability to retrieve the expected information and map their answers to the answer scale,…”
Banderas, B., Skup, M., Shields, A. L., Mazar, I., & Ganguli, A. (2017) · Current Medical Research and Opinion · doi:10.1080/03007995.2017.1338562 · Cited by 10 · see citing works
Miller, H., Lawson, D., Power, E., das Nair, R., Sathananthan, N., & Wong, D. (2022) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2022.01.003 · Cited by 10 · see citing works
Use of Computer and Cellular Phone Technology by Older Rural Adults
O’BRIEN, T. R., TREIBER, F., JENKINS, C., & MERCIER, A. (2014) · CIN: Computers, Informatics, Nursing · doi:10.1097/cin.0000000000000080 · Cited by 8 · see citing works
Tian, T., Sun, J., Jiang, Y., Guo, Q., Huang, Z., Wang, D., Rahman, A., Li, X., & Yang, L. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1168117 · Cited by 8 · see citing works
Tokgöz, P., Hafner, J., & Dockweiler, C. (2023) · Das Gesundheitswesen · doi:10.1055/a-2098-3108 · Cited by 5 · see citing works
- Authors
- Beatty, P. C., & Willis, G. B.
- Affiliations
- Centers for Disease Control and Prevention (US)
- Venue
- Public Opinion Quarterly
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-06-05
- Volume
- 71
- Issue
- 2
- Pages
- 287-311
- ISSN
- 0033-362X (ISSN-L)
- 1537-5331
- OpenAlex
- W2096834156
- Cited by
- 1,615
- References
- 37
- Citations by year
- 2026: 145 · 2025: 128 · 2024: 187 · 2023: 138 · 2022: 133 · 2021: 149 · 2020: 104 · 2019: 93 · 2018: 84 · 2017: 82 · 2016: 75 · 2015: 68 · 2014: 66 · 2013: 43 · 2012: 40
- FWCI
- 52.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Public Opinion Quarterly (publishedVersion)
- Funders
- National Institutes of Health
- Centers for Disease Control and Prevention
- National Cancer Institute
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- cognitive interviewing, questionnaire development, questionnaire evaluation, probe selection, interviewer role, evidence standards, best practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
