CoChat
Sign Up
Cited by 72Computational and Text Analysis MethodsQualitative Research Methods and ApplicationsData Visualization and Analytics

PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis

Open access
Gebreegziabher, Simret Araya, Zhang, Zheng, Tang, Xiaohang, Meng, Yihao, Glassman, Elena L., Li, Toby Jia-Jun·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3581352
Open this paper in CoChat — free accountOpen PDF

Abstract

Over the years, the task of AI-assisted data annotation has seen remarkable advancements. However, a specific type of annotation task, the qualitative coding performed during thematic analysis, has characteristics that make effective human-AI collaboration difficult. Informed by a formative study, we designed PaTAT, a new AI-enabled tool that uses an interactive program synthesis approach to learn flexible and expressive patterns over user-annotated codes in real-time as users annotate data. To accommodate the ambiguous, uncertain, and iterative nature of thematic analysis, the use of user-interpretable patterns allows users to understand and validate what the system has learned, make direct fixes, and easily revise, split, or merge previously annotated codes. This new approach also helps human users to learn data characteristics and form new theories in addition to facilitating the “learning” of the AI model. PaTAT’s usefulness and effectiveness were evaluated in a lab user study.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3581352

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Educational AI systems integrate curriculum-bound reactive and proactive mechanisms.

    “Educational AI systems integrate curriculum-bound reactive and proactive mechanisms.”

    From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [59]

    The paper’s Introduction and §4.2.1 describe an AI assistant for qualitative coding, not an educational AI system. Section 6.4 mentions intelligent tutors only as a possible domain for applying its design strategies. Searches for curriculum-bound, reactive, and proactive mechanisms found no passage addressing the claim; page numbers were not available in the indexed passages. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 48 · Cited by 72

Research works printed but not linked: 19

  1. [2]2. Tehmina Basit. 2003. Manual or electronic? The role of coding in qualitative data analysis. Educational research 45, 2 (2003), 143-154.
  2. [5]5. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77-101.
  3. [7]7. Nan-chen Chen, Rafal Kocielnik, Margaret Drouhard, Vanessa Peña-Araya, Jina Suh, Keting Cen, Xiangyi Zheng, and Cecilia R Aragon. 2016. Challenges of applying machine learning to qualitative coding. In ACM SIGCHI Workshop on Human-Centered Machine Learning.
  4. [14]14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
  5. [20]20. Monica M Gonzalez. 2016. The coding manual for qualitative research: A review. The Qualitative Report 21, 8 (2016), 1546-1549.
  6. [25]25. Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. Program synthesis. Foundations and Trends® in Programming Languages 4, 1-2 (2017), 1-119.
  7. [28]28. Holtzblatt Karen and Jones Sandra. 2017. Contextual inquiry: A participatory technique for system design. In Participatory design. CRC Press, 177-210.
  8. [31]31. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann, Chichester, West Sussex, U.K.
  9. [32]32. Vu Le, Daniel Perelman, Oleksandr Polozov, Mohammad Raza, Abhishek Udupa, and Sumit Gulwani. 2017. Interactive program synthesis. arXiv preprint arXiv:1703.03539 (2017). · arXiv
  10. [34]34. Laurent Letourneau-Guillon, David Camirand, Francois Guilbert, and Reza Forghani. 2020. Artifcial intelligence applications for workfow, process op- timization and predictive analytics. Neuroimaging Clinics 30, 4 (2020), e1-e15.
  11. [41]41. Megh Marathe and Kentaro Toyama. 2018. Semi-automated coding for qualitative research: A user-centered inquiry and initial prototypes. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1-12.
  12. [44]44. Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. 2019. A BERT-based transfer learning approach for hate speech detection in online social media. In International Conference on Complex Networks and Their Applications. Springer, 928-940.
  13. [48]48. Lawrence A Palinkas, Sarah M Horwitz, Carla A Green, Jennifer P Wisdom, Naihua Duan, and Kimberly Hoagwood. 2015. Purposeful sampling for quali- tative data collection and analysis in mixed method implementation research. Administration and policy in mental health and mental health services research 42, 5 (2015), 533-544.
  14. [50]50. Sarvapali D Ramchurn, Sebastian Stein, and Nicholas R Jennings. 2021. Trust- worthy human-AI partnerships. Iscience 24, 8 (2021), 102891.
  15. [53]53. Cynthia Rudin. 2015. Can machine learning be useful for social science. The Cities: An essay collection from the Decent City initiative 9 (2015), 86-90.
  16. [60]60. David Wicks. 2017. The coding manual for qualitative researchers. Qualitative research in organizations and management: an international journal (2017).
  17. [61]61. Charlene A Winters, Shirley Cudney, and Therese Sullivan. 2010. The Evolution of a Coding Schema in a Paced Program of Research. Qualitative Report 15, 6 (2010), 1415-1430.
  18. [66]66. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. Webanno: A fexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 1-6.
  19. [70]70. Sue Ziebland and Ann McPherson. 2006. Making sense of qualitative data analysis: an introduction with illustrations from DIPEx (personal experiences of health and illness). Medical education 40, 5 (2006), 405-414.

Authors
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J.-J.
Affiliations
  • University of Notre Dame (US)
  • University of Liverpool (GB)
  • Institution of Artificial Intelligence and Robotics, China
  • Harvard University (US)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-19
DOI
10.1145/3544548.3581352
OpenAlex
W4366548345
Semantic Scholar
8c5d8d674d0274f56658dfdd6891b399694255d1
Cited by
72
References
48
Citations by year
2026: 16 · 2025: 26 · 2024: 21 · 2023: 9
FWCI
37.45
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Funders
NSF (National Science Foundation) (2107391, 2123965)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Education › Qualitative Research Methods and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
Keywords
human-AI collaboration, qualitative coding, thematic analysis, explainable AI, rule synthesis, code annotation, iterative annotation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review