CoChat

Content moderation by LLM: from accuracy to legitimacy

Open access
Huang, Tao · 2025 · Artificial Intelligence Review

The DOI matches its Crossref record; 124 of the 124 references it cites were found in a registry or an index.

Abstract

One trending application of LLM (large language model) is to use it for content moderation in online platforms. Most current studies on this application have focused on the metric of accuracy —the extent to which LLMs make correct decisions about content. This article argues that accuracy is insufficient and misleading because it fails to grasp the distinction between easy cases and hard cases, as well as the inevitable trade-offs in achieving higher accuracy. Closer examination reveals that content moderation is a constitutive part of platform governance, the key to which is to gain and enhance legitimacy . Instead of making moderation decisions correctly, the chief goal of LLMs is to make them legitimate. In this regard, this article proposes a paradigm shift from the single benchmark of accuracy towards a legitimacy-based framework for evaluating the performance of LLM moderators. The framework suggests that for easy cases, the key is to ensure accuracy, speed, and transparency, while for hard cases, what matters is reasoned justification and user participation. Examined under this framework, LLMs’ real potential in moderation is not accuracy improvement. Rather, LLMs can better contribute in four other aspects: to conduct screening of hard cases from easy cases, to provide quality explanations for moderation decisions, to assist human reviewers in getting more contextual information, and to facilitate user participation in a more interactive way. To realize these contributions, this article proposes a workflow for incorporating LLMs into the content moderation system. Using normative theories from law and social sciences to critically assess the new technological application, this article seeks to redefine LLMs’ role in content moderation and redirect relevant research in this field.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10462-025-11328-1

Cites

124 of 124 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 A proposed workflow of incorporating LLM into content moderation. As shown in the figure, LLM can contribute in at least 6 occasions (numbers in red box): 1. distinguishing easy cases and hard cases; 2. providing information to human moderators; 3. helping human moderators generate justification for the moderation; 4. receiving feedback or appeal request from users; 5. assisting experts review by providing information; 6. disclosing results and justifications to users in an interactive way
Fig. 1 A proposed workflow of incorporating LLM into content moderation. As shown in the figure, LLM can contribute in at least 6 occasions (numbers in red box): 1. distinguishing easy cases and hard cases; 2. providing information to human moderators; 3. helping human moderators generate justification for the moderation; 4. receiving feedback or appeal request from users; 5. assisting experts review by providing information; 6. disclosing results and justifications to users in an interactive wayCC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 124 references

Research works printed but not linked: 27

  1. Bonaventura CD et al (2025) From detection to explanation: effective learning strategies for LLMs in online abusive language research. In: Proceedings of the 31st international conference on computational linguistics. Association for computational linguistics, pp 2067–2084. https://aclanthology.org/2025.coling-main.141/
  2. Bonaventura CD et al (2024) Is explanation all you need? An expert survey on LLM-generated explanations for abusive language detection. In: Tenth Italian conference on computational linguistics (CLiC-it 2024). https://kclpure.kcl.ac.uk/portal/en/publications/is-explanation-all-you-need-an-expert-survey-on-llm-generated-exp
  3. Yang D et al (2024) On verbalized confidence scores for LLMs. Preprint at arXiv:https://arxiv.org/abs/2412.14737 · arXiv
  4. Villate-Castillo G et al (2024) A collaborative content moderation framework for toxicity detection based on conformalized estimates of annotation disagreement. Preprint at arXiv:https://arxiv.org/abs/2411.04090v2 · arXiv
  5. Guerdan L et al (2024) A framework for evaluating LLMs under task indeterminacy. Preprint at arXiv:https://arxiv.org/abs/2411.13760 · arXiv
  6. Xiong M et al (2024) Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In: The 12th international conference on learning representations. https://openreview.net/forum?id=gjeQKFxFpZ
  7. Sharma M et al (2023) Towards understanding sycophancy in language models. Preprint at arXiv:https://arxiv.org/abs/2310.13548 · arXiv
  8. Kroeger N et al (2024) Are large language models post hoc explainers? Preprint at arXiv:https://arxiv.org/abs/2310.05797v3 · arXiv
  9. Manvi R et al (2024) Large language models are geographically biased. In: Proceedings of the 41st international conference on machine learning. ICML’24. Vienna, Austria
  10. Wang Y et al (2024) ImpScore: a learnable metric for quantifying the implicitness level of language. Preprint at arXiv:https://arxiv.org/abs/2411.05172v2 · arXiv
  11. Wang Z et al (2024) MLLM-as-a-judge for image safety without human labeling. Preprint at arXiv:https://arxiv.org/abs/2501.00192 · arXiv
  12. Bloch-Wehba H (2019) Global platform governance: private power in the shadow of the state. SMU Law Rev 72:27–80
  13. Boicel A (2024) Using LLMs to moderate content: are they ready for commercial use? https://www.techpolicy.press/using-llms-to-moderate-content-are-they-ready-for-commercial-use/. Accessed 13 Aug 2024
  14. Brennan-Marquez K (2017) Plausible cause: explanatory standards in the age of powerful machines. Vanderbilt Law Rev 70:1249–1301
  15. Divakaran DM, Peddinti ST (2024) LLMs for cyber security: new opportunities. Preprint at arXiv:https://arxiv.org/abs/2404.11338 · arXiv
  16. Elkin-Koren N, Perel M (2023) Speech contestation by design: democratizing speech governance by AI. Fla State Univ Law Rev 50:611–665
  17. Freeman J (2000) The private role in public governance. New York Univ Law Rev 75:543–675
  18. Hutchinson AC (1989) Democracy and determinacy: an essay on legal interpretation. Univ Miami Law Rev 43:541–576
  19. Kaminski ME (2019) Binary governance: lessons from the GDPR’s approach to algorithmic accountability. Southern California Law Rev 92:1529–1616
  20. Klonick K (2018) The new governors: the people, rules, and processes governing online speech. Harv Law Rev 131:1598–1670
  21. Meta (2022). How Meta prioritizes content for review. https://transparency.meta.com/policies/improving/prioritizing-content-review/. Accessed 13 Aug 2024
  22. Meta (2022). How review teams work. https://transparency.meta.com/enforcement/detecting-violations/how-review-teams-work/. Accessed 13 Aug 2024
  23. Nahmias Y, Perel M (2021) The oversight of content moderation by AI: impact assessments and their limitations. Harv J Legislat 58:145–194
  24. Schauer F (1985) Easy cases. Southern California Law Rev 58:399–440
  25. Solum LB (2004) Procedural justice. Southern California Law Rev 78:181–321
  26. Suzor N et al (2019) What do we mean when we talk about transparency? Toward meaningful transparency in commercial content moderation. Int J Commun 13:1526–1543
  27. Willner D, Chakrabarti S (2024) Using LLMs for policy-driven content classification. https://www.techpolicy.press/using-llms-for-policy-driven-content-classification/. Accessed 13 Aug 2024

Cited by

See all 24

Metadata

Authors
Huang, T.
Affiliations
City University of Hong Kong (HK)
Corresponding author
Huang, Tao
Venue
Artificial Intelligence Review
Published
2025-07-19
Volume
58
Issue
10
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4412962348
Cited by
24
References
80
Citations by year
2026: 19 · 2025: 4
FWCI
24.05
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence Review (publishedVersion, cc-by-nc-nd)
  • CityU Scholars (submittedVersion, cc-by-nc-nd)
Funders
City University of Hong Kong (7200786)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Freedom of Expression and Defamation
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
content moderation, large language models, platform governance, legitimacy, explainable AI, user participation, digital platforms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Content moderation by LLM: from accuracy to legitimacy. https://app.cochat.ai/papers/10.1007/s10462-025-11328-1
The paper
Huang, T. (2025). Content moderation by LLM: from accuracy to legitimacy. Artificial Intelligence Review. https://doi.org/10.1007/s10462-025-11328-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.