CoChat

“Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics

Elrod, Nathan J., Kropczynski, Jess, Grace, Rob · 2025 · 2025 IEEE World Forum on Public Safety Technology (WF-PST)

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.

Abstract

This study introduces a novel extension of Krippendorff's set-based “multi-value” inter-coder analysis, eliminating the need for coder adjudication by evaluating the agreement of multiset sentiment responses. We use open-source large language models in an HPC environment to perform sentiment analysis on COVID-19 social media data, performing queries multiple times per-text, per-model; a set of human coder sentiment is included as well. We evaluate reliability through internal consistency (Shannon entropy) with respect to each coder, and validity by comparing coder response multisets to one another via Krippendorff's alpha and a Dice-Sørensen metric. LLMs showed higher levels of internal agreement than human coders, with all coders showing low multiset (“inter-inter-coder”) agreement between one another.

Overview

Integrity signals

Registry

Crossref · 10.1109/wf-pst65083.2025.00017

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 62 references

Research works printed but not linked: 8

  1. Elrod, 2021, An exploration of methods using social media to examine local attitudes towards mask-wearing during a pandemic
  2. Touvron, 2023, Llama: Open and efficient foundation language models
  3. Galton, 1892, Finger Prints. MacMillan and Co.
  4. Ramamurthy, 2023, Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization
  5. Kristensen-McLachlan, 2023, Chatbots are not reliable text annotators
  6. Salton, 1983, Text Analysis and Automatic Indexing.
  7. Wang, 2023, Self-consistency improves chain of thought reasoning in language models
  8. Deb, 2024, Fill in the blank: Exploring and enhancing 11 m capabilities for backward reasoning in math word problems

Cited by

Still being checked.

Metadata

Authors
Elrod, N. J., Kropczynski, J., & Grace, R.
Affiliations
University of Cincinnati (US)
Venue
2025 IEEE World Forum on Public Safety Technology (WF-PST)
Publisher
IEEE
Published
2025-09-23
Pages
66-73
OpenAlex
W7133532271
Cited by
0
References
53
FWCI
0.00
Type
conference paper
Language
en
Open access
closed
Locations
IEEE World Forum on Public Safety Technology (WF-PST) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
intercoder reliability, large language models, Krippendorff's alpha, sentiment analysis, Shannon entropy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: “Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics. https://app.cochat.ai/papers/10.1109/wf-pst65083.2025.00017
The paper
Elrod, N. J., Kropczynski, J., & Grace, R. (2025). “Inter-Inter-Coder” Analysis: A Multiset Approach for LLM Evaluation in Public Safety Analytics. 2025 IEEE World Forum on Public Safety Technology (WF-PST). https://doi.org/10.1109/wf-pst65083.2025.00017

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.