CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Collaborative Qualitative Analysis (CQA) can enhance qualitative analysis rigor and depth by incorporating varied viewpoints. Nevertheless, ensuring a rigorous CQA procedure itself can be both complex and costly. To lower this bar, we take a theoretical perspective to design a one-stop, end-to-end workflow, CollabCoder, that integrates Large Language Models (LLMs) into key inductive CQA stages. In the independent open coding phase, CollabCoder offers AI-generated code suggestions and records decision-making data. During the iterative discussion phase, it promotes mutual understanding by sharing this data within the coding team and using quantitative metrics to identify coding (dis)agreements, aiding in consensus-building. In the codebook development phase, CollabCoder provides primary code group suggestions, lightening the workload of developing a codebook from scratch. A 16-user evaluation confirmed the effectiveness of CollabCoder, demonstrating its advantages over the existing CQA platform. All related materials of CollabCoder, including code and further extensions, will be included in: https://gaojie058.github.io/CollabCoder/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Literature review as a research methodology: An overview and guidelines
Snyder, H. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.07.039 · Cited by 11,524
Grounded theory research: Procedures, canons, and evaluative criteria
Corbin, J. M., & Strauss, A. (1990) · Qualitative Sociology · doi:10.1007/bf00988593 · Cited by 11,133
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“Therefore, preserving the enjoyment of qualitative research and designing for appropriate reliance [44] to avoid mis-use [19] or over-trust can be a complex challenge [67].”
Clark, H. H., & Brennan, S. E. (1991) · Perspectives on socially shared cognition. · doi:10.1037/10096-006 · Cited by 4,555
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871
The interdisciplinary study of coordination
Malone, T. W., & Crowston, K. (1994) · ACM Computing Surveys · doi:10.1145/174666.174668 · Cited by 3,504 · cited 3×
“Similarly, shared mental model is a conception in team coordination theory [22, 47].”
Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars.
Maguire, M., & Delahunt, B. (2017) · All Ireland Journal of Higher Education · doi:10.62707/aishej.v9i3.335 · Cited by 2,099
Olson, G. M., & Olson, J. S. (2000) · Human–Computer Interaction · doi:10.1207/s15327051hci1523_4 · Cited by 1,983 · cited 1×
“With the aforementioned design goals in mind, we finalized the CollabCoder system and its CQA workflow (refer to Figure 3).”
MethodIssues of validity and reliability in qualitative research
Noble, H., & Smith, J. (2015) · Evidence Based Nursing · doi:10.1136/eb-2015-102054 · Cited by 1,758
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 2×
“We took note of the role LLMs undertook during these discussions.”
BackgroundMethodMcDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Allen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442
Chen, N.-C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185515 · Cited by 152
Does Distance Still Matter? Revisiting the CSCW Fundamentals on Distributed Collaboration
Bjørn, P., Esbensen, M., Jensen, R. E., & Matthiesen, S. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2670534 · Cited by 147
Gilbert, L. S., Jackson, K., & di Gregorio, S. (2014) · Handbook of Research on Educational Communications and Technology · doi:10.1007/978-1-4614-3185-5_18 · Cited by 73
Jiang, J. A., Wade, K., Fiesler, C., & Brubaker, J. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449168 · Cited by 1
Cited by
Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
Szymanski, A., Ziems, N., Eicher-Miller, H. A., Li, T. J.-J., Jiang, M., & Metoyer, R. A. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712091 · Cited by 77
Self-citationMao, R., Gu, X., & Gu, Y. (2026) · Current Issues in Tourism · doi:10.1080/13683500.2026.2722194 · Cited by 2
Metadata
- Authors
- Gao, J., Guo, Y., Lim, G., Zhang, T., Zhang, Z., Li, T. J.-J., & Perrault, S. T.
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-29
- OpenAlex
- W4396833424
- Cited by
- 73
- References
- 18
- FWCI
- 76.89
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.