Why Open-Ended Survey Questions Are Unlikely to Support Rigorous Qualitative Insights
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Health professions education researchers are increasingly relying on a combination of quantitative and qualitative research methods to explore complex questions in the field. This important and necessary development, however, creates new methodological challenges that can affect both the rigor of the research process and the quality of the findings. One example is “qualitatively” analyzing free-text responses to survey or assessment instrument questions. In this Invited Commentary, the authors explain why analysis of such responses rarely meets the bar for rigorous qualitative research. While the authors do not discount the potential for free-text responses to enhance quantitative findings or to inspire new research questions, they caution that these responses rarely produce data rich enough to generate robust, stand-alone insights. The authors consider exemplars from health professions education research and propose strategies for treating free-text responses appropriately.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601
Strategies for ensuring trustworthiness in qualitative research projects
Shenton, A. K. (2004) · Education for Information · doi:10.3233/efi-2004-22201 · Cited by 9,748
Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research
Tracy, S. J. (2010) · Qualitative Inquiry · doi:10.1177/1077800410383121 · Cited by 7,536
Enhancing the quality and credibility of qualitative analysis.
(1999) · PubMed Central · Cited by 4,133
Critical analysis of strategies for determining rigor in qualitative inquiry
Morse, J. M. (2015) · Qualitative Health Research · doi:10.1177/1049732315588501 · Cited by 2,809
Pillow, W. (2003) · International Journal of Qualitative Studies in Education · doi:10.1080/0951839032000060635 · Cited by 2,093
Emerging Criteria for Quality in Qualitative and Interpretive Research
Lincoln, Y. S. (1995) · Qualitative Inquiry · doi:10.1177/107780049500100301 · Cited by 1,568
Discourse and Text: Linguistic and Intertextual Analysis within Discourse Analysis
Fairclough, N. (1992) · Discourse & Society · doi:10.1177/0957926592003002004 · Cited by 1,282
"Any other comments?" Open questions on questionnaires – a bane or a bonus to research?
O'Cathain, A., & Thomas, K. J. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-25 · Cited by 646
Cited by
What makes for effective feedback: staff and student perspectives
Dawson, P., Henderson, M., Mahoney, P., Phillips, M., Ryan, T., Boud, D., & Molloy, E. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1467877 · Cited by 514
Beyond the guise of saturation: rigor and qualitative interview data
LaDonna, K. A., Artino, A. R., & Balmer, D. F. (2021) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-21-00752.1 · Cited by 457
Self-citationThe challenges of feedback in higher education
Henderson, M., Ryan, T., & Phillips, M. (2019) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2019.1599815 · Cited by 335
Comparing Generative AI and teacher feedback: student perceptions of usefulness and trustworthiness
Henderson, M., Bearman, M., Chung, J., Fawns, T., Buckingham Shum, S., Matthews, K. E., & de Mello Heredia, J. (2026) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2502582 · Cited by 211
“Such an approach was considered to be most appropriate given the broad nature of the comments and the recognised complications of coding latent meanings from short open-ended survey responses (LaDonna, Taylor, and Lingard 2018).”
Burnout and resilience among Canadian palliative care physicians
Wang, C., Grassau, P., Lawlor, P. G., Webber, C., Bush, S. H., Gagnon, B., Kabir, M., & Spilg, E. G. (2020) · BMC Palliative Care · doi:10.1186/s12904-020-00677-z · Cited by 60
Metadata
- Authors
- LaDonna, K. A., Taylor, T., & Lingard, L.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-03
- Volume
- 93
- Issue
- 3
- Pages
- 347-349
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W2773586704
- Cited by
- 295
- References
- 12
- FWCI
- 34.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why Open-Ended Survey Questions Are Unlikely to Support Rigorous Qualitative Insights. https://app.cochat.ai/papers/10.1097/acm.0000000000002088
- The paper
- LaDonna, K. A., Taylor, T., & Lingard, L. (2018). Why Open-Ended Survey Questions Are Unlikely to Support Rigorous Qualitative Insights. Academic Medicine. https://doi.org/10.1097/acm.0000000000002088
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.