Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 3×
“Few-shots: Since recent work showed conflicting results on the number of examples in a prompt [8], we explored different prompt settings.”
BackgroundInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 1×
“For example, LLMs could perform classification tasks if the prompt specifies a set of labels as the output space [4].”
BackgroundWordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374 · cited 1×
“, GPT-3 [2], PaLM [3], OPT [16]) offers new capabilities, including creative writing [14], programming 1, etc.”
BackgroundKorngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · cited 1×
“Although recent studies have demonstrated LLMs utility in many domains, studies have shown LLMs are error-prone and have limited capability in capturing language structure and nuanced meanings [6], which are crucial in qualitative analysis.”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124
Conversational agents for fostering curiosity-driven learning in children
Abdelghani, R., Oudeyer, P.-Y., Law, E., de Vulpillières, C., & Sauzéon, H. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102887 · Cited by 39
Self-citation
Research works printed but not linked: 5
- 10.3115/v1/W14-2513 · DOI · Paper page
- 10.1145/2957276.2957280 · DOI · Paper page
- 10.1145/2998181.2998363 · DOI
- 10.1145/3411764.3445591 · DOI · Paper page
- Susan Zhang , Stephen Roller , Naman Goyal , Mikel Artetxe , Moya Chen , Shuohui Chen , Christopher Dewan , Mona Diab , Xian Li , Xi Victoria Lin , 2022 . Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068(2022). Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068(2022). · arXiv
Cited by
An Examination of the Use of Large Language Models to Aid Analysis of Textual Data
(2024) · doi:10.1177/16094069241231168 · Cited by 227
Yang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L.-W. (2023) · arXiv · doi:10.48550/arxiv.2310.15100 · Cited by 110
Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., & Majumdar, A. (2026) · ACM Computing Surveys · doi:10.1145/3744238 · Cited by 61
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
Gao, J., Gebreegziabher, S. A., Choo, K. T. W., Li, T. J.-J., Perrault, S. T., & Malone, T. W. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650786 · Cited by 57
“Similarly, example-based prompting entails providing LLMs with a handful of relevant examples that clearly demonstrate the expected input and output [76], with the LLM tasked to independently discern the underlying rationale.”
Embedding Democratic Values into Social Media AIs via Societal Objective Functions
Jia, C., Lam, M. S., Mai, M. C., Hancock, J. T., & Bernstein, M. S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641002 · Cited by 31
ContextQ: Generated Questions to Support Meaningful Parent-Child Dialogue While Co-Reading
Dietz Smith, G., Prasad, S., Davidson, M. J., Findlater, L., & Shapiro, R. B. (2024) · Proceedings of the 23rd Annual ACM Interaction Design and Children Conference · doi:10.1145/3628516.3655809 · Cited by 29
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Shaping the Emerging Norms of Using Large Language Models in Social Computing Research
Shen, H., Li, T., Li, T. J.-J., Park, J. S., & Yang, D. (2023) · Computer Supported Cooperative Work and Social Computing · doi:10.1145/3584931.3606955 · Cited by 21
“For instance, LLM-powered tools have been shown to assist researchers to more efficiently analyze textual data [17, 18], replicate social science studies [3], and enable new ways to prototype emergent social dynamics in social computing systems where it is infeasible or dangerous to conduct in the wild studies [14].”
BackgroundCan GPT-4 Replicate Empirical Software Engineering Research?
Liang, J. T., Badea, C., Bird, C., DeLine, R., Ford, D., Forsgren, N., & Zimmermann, T. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660767 · Cited by 19
“We then sorted by citation and selected the first 20 results to limit the search results.”
MethodChan, S., Li, J., Yao, B., Mahmood, A., Huang, C.-M., Jimison, H., Mynatt, E. D., & Wang, D. (2023) · arXiv · doi:10.48550/arxiv.2310.05853 · Cited by 16
Silent guardian: protecting text from malicious exploitation by large language models
Zhao, J., Chen, K., Yuan, X., Qi, Y., Zhang, W., & Yu, N. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3455775 · Cited by 12
Self-citationMao, R., Gu, X., & Gu, Y. (2026) · Current Issues in Tourism · doi:10.1080/13683500.2026.2722194 · Cited by 2
Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction
Kim, H., Choi, Y., Yang, T., Lee, H., Park, C., Lee, Y., Kim, J. Y., & Kim, J. (2024) · arXiv · doi:10.48550/arxiv.2407.13166
“Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics
Elrod, N. J., Kropczynski, J., & Grace, R. (2025) · 2025 IEEE World Forum on Public Safety Technology (WF-PST) · doi:10.1109/wf-pst65083.2025.00017
Metadata
- Authors
- Xiao, Z., Yuan, X., Liao, Q. V., Abdelghani, R., & Oudeyer, P.-Y.
- Venue
- 28th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2023-03-27
- Pages
- 75-78
- OpenAlex
- W4360978668
- Cited by
- 230
- References
- 11
- FWCI
- 122.41
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.