Annotating social media data from vulnerable populations: evaluating disagreement between domain experts and graduate student annotators
Open accessAbstract
Researchers in computer science have spent considerable time developing methods to increase the accuracy and richness of annotations.However, there is a dearth in research that examines the positionality of the annotator, how they are trained and what we can learn from disagreements between different groups of annotators.In this study, we use qualitative analysis, statistical and computational methods to compare annotations between Chicago-based domain experts and graduate students who annotated a total of 1,851 tweets with images that are a part of a larger corpora associated with the Chicago Gang Intervention Study, which aims to develop a computational system that detects aggression and loss among gang-involved youth in Chicago.We found evidence to support the study of disagreement between annotators and underscore the need for domain expertise when reviewing Twitter data from vulnerable populations.Implications for annotation and content moderation are discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The characteristics of those that do the annotations can significantly influence how they annotate.
“for many annotations tasks, the characteristics of those that do the annotations can significantly influence how they annotate (Olteanu et al., 2017a; Patton et al., 2019).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Patton et al., 2019
There are known differences in how non-experts and experts interact with and validate systems outputs, particularly for critical application domains such as health.
“This is problematic as there are known differences in how non-experts and experts interact with and validate systems outputs (White et al., 2009; Patton et al., 2019), particularly for critical application domains such as health.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Patton et al., 2019
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Critical technocultural discourse analysis
Brock, A. (2018) · New Media & Society · doi:10.1177/1461444816677532 · Cited by 318 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“This is problematic as there are known differences in how non-experts and experts interact with and validate systems outputs (White et al., 2009; Patton et al., 2019), particularly for critical application domains such as health.”
BackgroundFrom Bias to Repair: Error as a Site of Collaboration and Negotiation in Applied Data Science Work
Lin, C. K., & Jackson, S. J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579607 · Cited by 22 · see citing works
- Authors
- Patton, D., Blandfort, P., Frey, W., Gaskell, M., & Karaman, S.
- Affiliations
- Columbia University (US)
- German Research Centre for Artificial Intelligence (DE)
- Venue
- Proceedings of the Annual Hawaii International Conference on System Sciences
- Published
- 2019
- ISSN
- 1530-1605 (ISSN-L)
- 2572-6862
- OpenAlex
- W2926526813
- Cited by
- 23
- References
- 2
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 1 · 2023: 4 · 2022: 3 · 2021: 2 · 2020: 3 · 2019: 2 · 2018: 1
- FWCI
- 0.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Hawaii International Conference on System Sciences (HICSS) (publishedVersion, cc-by-nc-nd)
- Journal of the Association for Information Systems (submittedVersion)
- ScholarSpace (University of Hawaii at Manoa) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- annotator disagreement, domain knowledge, vulnerable populations, Twitter, aggression detection, loss detection, content moderation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
