“Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics
The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
This study introduces a novel extension of Krippendorff's set-based “multi-value” inter-coder analysis, eliminating the need for coder adjudication by evaluating the agreement of multiset sentiment responses. We use open-source large language models in an HPC environment to perform sentiment analysis on COVID-19 social media data, performing queries multiple times per-text, per-model; a set of human coder sentiment is included as well. We evaluate reliability through internal consistency (Shannon entropy) with respect to each coder, and validity by comparing coder response multisets to one another via Krippendorff's alpha and a Dice-Sørensen metric. LLMs showed higher levels of internal agreement than human coders, with all coders showing low multiset (“inter-inter-coder”) agreement between one another.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Sentiment Analysis and Opinion Mining
Liu, B. (2012) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02145-9 · Cited by 3,281
Reliability of Content Analysis: The Case of Nominal Scale Coding
Scott, W. A. (1955) · Public Opinion Quarterly · doi:10.1086/266577 · Cited by 1,853
Eliminative Materialism and Propositional Attitudes
Churchland, P. M., & Journal of Philosophy Inc. (1981) · Journal of Philosophy · doi:10.5840/jphil198178268 · Cited by 1,814
Research works printed but not linked: 8
- Elrod, 2021, An exploration of methods using social media to examine local attitudes towards mask-wearing during a pandemic
- Touvron, 2023, Llama: Open and efficient foundation language models
- Galton, 1892, Finger Prints. MacMillan and Co.
- Ramamurthy, 2023, Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization
- Kristensen-McLachlan, 2023, Chatbots are not reliable text annotators
- Salton, 1983, Text Analysis and Automatic Indexing.
- Wang, 2023, Self-consistency improves chain of thought reasoning in language models
- Deb, 2024, Fill in the blank: Exploring and enhancing 11 m capabilities for backward reasoning in math word problems
Cited by
Still being checked.
Metadata
- Authors
- Elrod, N. J., Kropczynski, J., & Grace, R.
- Affiliations
- University of Cincinnati (US)
- Venue
- 2025 IEEE World Forum on Public Safety Technology (WF-PST)
- Publisher
- IEEE
- Published
- 2025-09-23
- Pages
- 66-73
- OpenAlex
- W7133532271
- Semantic Scholar
- 94e56117a759e6912d7b6a6f0a3928d15a644807
- Cited by
- 0
- References
- 53
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE World Forum on Public Safety Technology (WF-PST) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- intercoder reliability, large language models, Krippendorff's alpha, sentiment analysis, Shannon entropy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics. https://app.cochat.ai/papers/10.1109/wf-pst65083.2025.00017
- The paper
- Elrod, N. J., Kropczynski, J., & Grace, R. (2025). “Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics. 2025 IEEE World Forum on Public Safety Technology (WF-PST). https://doi.org/10.1109/wf-pst65083.2025.00017
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.