Qualitative Research Methods for Large Language Models: Conducting Semi-Structured Interviews with ChatGPT and BARD on Computer Science Education
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
In the current era of artificial intelligence, large language models such as ChatGPT and BARD are being increasingly used for various applications, such as language translation, text generation, and human-like conversation. The fact that these models consist of large amounts of data, including many different opinions and perspectives, could introduce the possibility of a new qualitative research approach: Due to the probabilistic character of their answers, “interviewing” these large language models could give insights into public opinions in a way that otherwise only interviews with large groups of subjects could deliver. However, it is not yet clear if qualitative content analysis research methods can be applied to interviews with these models. Evaluating the applicability of qualitative research methods to interviews with large language models could foster our understanding of their abilities and limitations. In this paper, we examine the applicability of qualitative content analysis research methods to interviews with ChatGPT in English, ChatGPT in German, and BARD in English on the relevance of computer science in K-12 education, which was used as an exemplary topic. We found that the answers produced by these models strongly depended on the provided context, and the same model could produce heavily differing results for the same questions. From these results and the insights throughout the process, we formulated guidelines for conducting and analyzing interviews with large language models. Our findings suggest that qualitative content analysis research methods can indeed be applied to interviews with large language models, but with careful consideration of contextual factors that may affect the responses produced by these models. The guidelines we provide can aid researchers and practitioners in conducting more nuanced and insightful interviews with large language models. From an overall view of our results, we generally do not recommend using interviews with large language models for research purposes, due to their highly unpredictable results. However, we suggest using these models as exploration tools for gaining different perspectives on research topics and for testing interview guidelines before conducting real-world interviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Guest, G., MacQueen, K., & Namey, E. (2012) · doi:10.4135/9781483384436 · Cited by 7,924
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · cited 1×
“This first version of GPT was trained on 4.5 GB of text from unpublished books [8].”
MethodInfluentialValidity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · cited 1×
“Large language models (LLMs) are artificial intelligence (AI) models based on deep learning (i.e., neural networks) used to generate text [3,4].”
BackgroundHow to use thematic analysis in qualitative research
Christou, P. A. (2022) · Journal of Qualitative Research in Tourism · doi:10.4337/jqrt.2023.0006 · Cited by 459
Laï, M.-C., Brian, M., & Mamzer, M.-F. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-019-02204-y · Cited by 294
Haan, M., Ongena, Y. P., Hommes, S., Kwee, T. C., & Yakar, D. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2018.12.043 · Cited by 141 · cited 1×
“Artificial intelligences and LLMs have not only been a topic of research since ChatGPT 3.5 and the associated breakthrough in public perception but already support numerous disciplines in practice, such as in medicine [22], healthcare [23], and economics [24].”
BackgroundInfluentialPlaying games with AIs: the limits of GPT-3 and similar large language models
Sobieszek, A., & Price, T. (2022) · Minds and Machines · doi:10.1007/s11023-022-09602-0 · Cited by 86
Cited by
Gupta, R., Nair, K., Mishra, M., Ibrahim, B., & Bhardwaj, S. (2024) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2024.100232 · Cited by 278
Zhang, X., Zhang, P., Shen, Y., Liu, M., Wang, Q., Gašević, D., & Fan, Y. (2024) · Frontiers of Digital Education · doi:10.1007/s44366-024-0028-5 · Cited by 83
Emerging Dynamics of ChatGPT in Academia: A Scoping Review
Mahrishi, M., Abbas, A., Radovanović, D., & Hosseini, S. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/b182ws13 · Cited by 28
Li, X., Li, T., Wang, M., Tao, S., Zhou, X., Wei, X., & Guan, N. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3539104 · Cited by 8
González-Cantera, T., & Bonacchi, C. (2025) · International Journal of Heritage Studies · doi:10.1080/13527258.2025.2543736 · Cited by 4
“Finally, we asked follow-up queries to explore in further depth any controversial or unclear points that emerged from ChatGPT-4o’s previous responses (Dengel et al. 2023).”
Metadata
- Authors
- Dengel, A., Gehrlein, R., Fernes, D., Görlich, S., Maurer, J., Pham, H. H., Großmann, G., & Eisermann, N. D. G.
- Venue
- Informatics
- Publisher
- MDPI AG
- Published
- 2023-10-12
- Volume
- 10
- Issue
- 4
- Pages
- 78
- ISSN
- 2227-9709 (ISSN-L)
- OpenAlex
- W4387573006
- Cited by
- 64
- References
- 9
- FWCI
- 30.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.