Using natural language processing technology for qualitative data analysis
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Social researchers often apply qualitative research methods to study groups and their communications artifacts. The use of computer-mediated communications has dramatically increased the volume of text available, but coding such text requires considerable manual effort. We discuss how systems that process text in human languages (i.e. natural language processing [NLP]) might partially automate content analysis by extracting theoretical evidence. We present a case study of the use of NLP for qualitative analysis in which the NLP rules showed good performance on a number of codes. With the current level of performance, use of an NLP system could reduce the amount of text to be examined by a human coder by an order of magnitude or more, potentially increasing the speed of coding by a comparable degree. The paper is significant as it is one of the first to demonstrate the use of high-level NLP techniques for qualitative data analysis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Qualitative Data Analysis: An Expanded Sourcebook
(1994) · Journal of Environmental Psychology · doi:10.1016/s0272-4944(05)80231-2 · Cited by 48,420 · cited 2×
“The result of the coding process is a text annotated (or tagged) with codes for the concepts exhibited (Miles & Huberman, 1994 ).”
BackgroundThe Presentation of Self in Everyday Life.
Naegele, K. D., & Goffman, E. (1956) · American Sociological Review · doi:10.2307/2089106 · Cited by 25,991
The Presentation of Self in Everyday Life
Goffman, E. (2023) · Social Theory Re-Wired · doi:10.4324/9781003320609-59 · Cited by 17,562
Politeness : Some Universals in Language Usage
Brown, P. (1987) · Cited by 12,120
Politeness: Some Universals in Language Usage
Brown, P., Levinson, S. C., & Gumperz, J. J. (1987) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813085 · Cited by 10,438
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education
Garrison, D. R., Anderson, T., & Archer, W. (1999) · The Internet and Higher Education · doi:10.1016/s1096-7516(00)00016-6 · Cited by 6,619 · cited 2×
“Garrison et al. (2000, p. 89) de fi ned social presence to be ‘ the ability of participants . . . to project their personal characteristics into the community, thereby presenting themselves to the other participants as “ real people ” .”
BackgroundMethodJurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Assessing Social Presence In Asynchronous Text-based Computer Conferencing
Rourke, L., Anderson, T. D., Garrison, D. R., & Archer, W. (1999) · AUSpace (Athabasca University) · Cited by 2,377
Qualitative Research in Information Systems
Myers, M. D. (1997) · MIS Quarterly · doi:10.2307/249422 · Cited by 2,365 · cited 2×
“But qualitative research can, in fact, adopt any research perspective: positivist, interpretivist or critical (Myers, 1997).”
BackgroundMethod
Cited by
Coding qualitative data: a synthesis guiding the novice
Skjott Linneberg, M., & Korsgaard, S. (2019) · Qualitative Research Journal · doi:10.1108/qrj-12-2018-0012 · Cited by 1,169
Industrial Artificial Intelligence in Industry 4.0 - Systematic Review, Challenges and Outlook
Peres, R. S., Jia, X., Lee, J., Sun, K., Colombo, A. W., & Barata, J. (2020) · IEEE Access · doi:10.1109/access.2020.3042874 · Cited by 693
Two Strategies for Qualitative Content Analysis: An Intramethod Approach to Triangulation
Renz, S. M., Carrington, J. M., & Badger, T. A. (2018) · Qualitative Health Research · doi:10.1177/1049732317753586 · Cited by 372
Integrating Machine Learning with Human Knowledge
Deng, C., Ji, X., Rainey, C., Zhang, J., & Lu, W. (2020) · iScience · doi:10.1016/j.isci.2020.101656 · Cited by 243
Chen, N.-C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185515 · Cited by 152
PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J.-J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581352 · Cited by 72
A qualitative analysis framework using natural language processing and graph theory
Tierney, P. (2012) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v13i5.1240 · Cited by 38
Fang, C., Markuzon, N., Patel, N., & Rueda, J.-D. (2022) · Value in Health · doi:10.1016/j.jval.2022.06.004 · Cited by 36
Augmenting assessment with AI coding of online student discourse: A question of reliability
Misiejuk, K., Kaliisa, R., & Scianna, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100216 · Cited by 33
Rigley, E., Bentley, C., Krook, J., & Ramchurn, S. D. (2024) · Global Policy · doi:10.1111/1758-5899.13299 · Cited by 31
“However, there are several drawbacks to NLP including the additional time-consuming steps of data preparation and rule development, and the loss of reliability in sparse data (Crowston et al., 2011).”
BackgroundRoles and politeness behavior in community-based free/libre open source software development
Wei, K., Crowston, K., Eseryel, U. Y., & Heckman, R. (2017) · Information & Management · doi:10.1016/j.im.2016.11.006 · Cited by 26
“Automatic analysis of thousands of email messages on linguistic symbols of politeness behaviors is still at its early development [41].”
BackgroundSelf-citationHuang, H.-W., Teng, D. C.-E., & Tiangco, J. A. N. Z. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10179-9 · Cited by 23
Metadata
- Authors
- Crowston, K., Allen, E. E., & Heckman, R.
- Venue
- International Journal of Social Research Methodology
- Publisher
- Informa UK Limited
- Published
- 2011-10-25
- Volume
- 15
- Issue
- 6
- Pages
- 523-543
- ISSN
- 1364-5579 (ISSN-L)
- 1464-5300
- OpenAlex
- W1981740417
- Cited by
- 170
- References
- 33
- FWCI
- 27.67
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.