From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis
Open accessAbstract
Subjective data annotation (SDA) plays an important role in many NLP tasks, including sentiment analysis, toxicity detection, and bias identification.Conventional SDA often treats annotator disagreement as noise, overlooking its potential to reveal deeper insights.In contrast, qualitative data analysis (QDA) explicitly engages with diverse positionalities and treats disagreement as a meaningful source of knowledge.In this position paper, we argue that human annotators are a key source of valuable interpretive insights into subjective data beyond surface-level descriptions.Through a comparative analysis of SDA and QDA methodologies, we examine similarities and differences in task nature (e.g., human's role, analysis content, cost, and completion conditions) and practice (annotation schema, annotation workflow, annotator selection, and evaluation).Based on this comparison, we propose five practical recommendations for enabling SDA to capture richer insights.We demonstrate these recommendations in a reinforcement learning from human feedback (RLHF) case study and envision that our interdisciplinary perspective will offer new directions for the field.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work has highlighted reflexive practices drawn from interpretivist traditions, such as encouraging data workers to reflect on their positionality and drawing on qualitative research methods to aid in data interpretation.
“Recent work has highlighted reflexive practices drawn from interpretivist traditions, such as encouraging data workers to reflect on their positionality [16] and drawing on qualitative research methods to aid in data interpretation [103].”
From Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter (2026) · cites this paper as [103]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 27
- Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal arabic irony corpus extracted from twitter. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 6265-6271.
- Desislava Aleksandrova, François Lareau, and Pierre André Ménard. 2019. Multilingual sentence- level bias detection in wikipedia. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 42-51.
- Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw, and Chris Biemann. 2023. Ex- ploring amharic hate speech data collection and clas- sification approaches. In Proceedings of the 14th international conference on recent advances in natu- ral language processing, pages 49-59.
- Dirk Berg-Schlosser. 2015. Comparative studies: Method and design. In James D. Wright, editor, In- ternational Encyclopedia of the Social & Behavioral Sciences (Second Edition), pages 439-444. Elsevier. · DOI
- Meghana Moorthy Bhat, Saghar Hosseini, Ahmed Has- san Awadallah, Paul Bennett, and Weisheng Li. 2021. Say 'YES' to positivity: Detecting toxic language in workplace communications. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 2017-2029, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
- Joachim Blatter, Markus Haverland, and Merlijn Van Hulst. 2016. Qualitative research in political science. Sage Publications Thousand Oaks.
- Isabel Cachola, Eric Holgate, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2018. Expressively vulgar: The socio-dynamics of vulgarity and its effects on sen- timent analysis in social media. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2927-2938.
- Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018a. Give me more feedback: Anno- tating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 621-631, Melbourne, Australia. Association for Computational Linguistics. · DOI
- Kathy Charmaz. 2005. Grounded theory in the 21st cen- tury: Applications for advancing social justice stud- ies. In Qualitative Research Conference, May, 2003, Carleton University, Ottawa, ON, Canada; Brief ex- cerpts from earlier drafts in a keynote address," Re- claiming Traditions and Re-forming Trends in Qual- itative Research," were presented at the aforemen- tioned conference and in a presentation," Suffering and the Self: Meanings of Loss in Chronic Illness," at the Sociology Department, University of California, Los Angeles, January 9, 2004. Sage Publications Ltd.
- Çagrı Çöltekin. 2020. A corpus of Turkish offensive language on social media. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 6174-6184, Marseille, France. European Language Resources Association.
- Jenny Fleming. 2018. Recognizing and resolving the challenges of being an insider researcher in work- integrated learning. International journal of work- integrated learning, 19(3):311-320.
- Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka- Wei Lee, and Simon Perrault. 2023. Coaicoder: Ex- amining the effectiveness of ai-assisted human-to- human collaboration in qualitative analysis. ACM Trans. Comput.-Hum. Interact. Just Accepted. · DOI
- Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Per- rault. 2024. Collabcoder: A lower-barrier, rigorous workflow for inductive collaborative qualitative anal- ysis with large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY, USA. Association for Computing Machinery. · DOI
- Barney Glaser and Anselm Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge.
- Sanjay Kairam and Jeffrey Heer. 2016. Parting crowds: Characterizing divergent interpretations in crowd- sourced annotation tasks. In Proceedings of the 19th ACM Conference on Computer-Supported Cooper- ative Work & Social Computing, CSCW '16, page 1637-1648, New York, NY, USA. Association for Computing Machinery. · DOI
- Tomoyuki Kajiwara, Chenhui Chu, Noriko Take- mura, Yuta Nakashima, and Hajime Nagahara. 2021. WRIME: A new dataset for emotional intensity es- timation with subjective and objective annotations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2095-2104, Online. Association for Computa- tional Linguistics. · DOI
- Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in quali- tative research: Norms and guidelines for cscw and hci practice. Proc. ACM Hum.-Comput. Interact., 3(CSCW). · DOI · Paper page
- Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A streamlined method for sourcing discourse- level argumentation annotations from the crowd. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1790-1796.
- Patrick Ngulube. 2015. Qualitative data analysis and interpretation: systematic search for meaning. Ad- dressing research challenges: making headway for developing researchers, 131(156):681-694.
- James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. " O'Reilly Me- dia, Inc.".
- Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 175-190, Seattle, United States. Association for Computational Linguistics. · DOI
- Johnny Saldaña. 2021. The coding manual for qualita- tive researchers. SAGE publications Ltd.
- Rickard Stureborg, Bhuwan Dhingra, and Jun Yang. 2023. Interface design for crowdsourcing hierarchi- cal multi-label text annotations. In Proceedings of the 2023 CHI Conference on Human Factors in Com- puting Systems, CHI '23, New York, NY, USA. As- sociation for Computing Machinery. · DOI
- Enrica Troiano, Carlo Strapparava, Gözde Özbal, and Serra Sinem Tekiroglu. 2018. A computational ex- ploration of exaggeration. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3296-3304, Brussels, Belgium. Association for Computational Linguistics. · DOI
- Carla Willig. 2012. Qualitative interpretation and anal- ysis in psychology. McGraw-Hill Education (UK).
- Carla Willig and Wendy Stainton Rogers. 2017. Inter- pretation in qualitative research. In Carla Willig and Wendy Stainton Rogers, editors, The SAGE Hand- book of Qualitative Research in Psychology, pages 274-288. SAGE Publications Ltd. · DOI
- Longyin Zhang, Xin Tan, Fang Kong, and Guodong Zhou. 2021. EDTC: A corpus for discourse-level topic chain parsing. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1304-1312, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. · DOI
Cited by
Sorted by citations
Tang, X., Li, J., & Wu, S. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790405 · Cited by 4 · see citing works
“While AI data annotation tasks have become increasingly interpretive [16, 85, 86, 103], most accessibility datasets are still annotated by data workers without relevant experience or expertise in disabilities to properly interpret the data.”
- Authors
- Wan, R., Wang, H., Huang, T.-H. K., & Gao, J.
- Affiliations
- Pennsylvania State University (US)
- Johns Hopkins University (US)
- Venue
- Proceedings of the Fourth Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP)
- Published
- 2025
- Pages
- 240-254
- OpenAlex
- W4416034494
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Education › Qualitative Research Methods and Applications
- Keywords
- qualitative data analysis, annotator disagreement, annotation workflow, annotation evaluation, reinforcement learning from human feedback, sentiment analysis, toxicity detection, bias detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
