Content moderation by LLM: from accuracy to legitimacy
Open accessThe DOI matches its Crossref record; 124 of the 124 references it cites were found in a registry or an index.
Abstract
One trending application of LLM (large language model) is to use it for content moderation in online platforms. Most current studies on this application have focused on the metric of accuracy —the extent to which LLMs make correct decisions about content. This article argues that accuracy is insufficient and misleading because it fails to grasp the distinction between easy cases and hard cases, as well as the inevitable trade-offs in achieving higher accuracy. Closer examination reveals that content moderation is a constitutive part of platform governance, the key to which is to gain and enhance legitimacy . Instead of making moderation decisions correctly, the chief goal of LLMs is to make them legitimate. In this regard, this article proposes a paradigm shift from the single benchmark of accuracy towards a legitimacy-based framework for evaluating the performance of LLM moderators. The framework suggests that for easy cases, the key is to ensure accuracy, speed, and transparency, while for hard cases, what matters is reasoned justification and user participation. Examined under this framework, LLMs’ real potential in moderation is not accuracy improvement. Rather, LLMs can better contribute in four other aspects: to conduct screening of hard cases from easy cases, to provide quality explanations for moderation decisions, to assist human reviewers in getting more contextual information, and to facilitate user participation in a more interactive way. To realize these contributions, this article proposes a workflow for incorporating LLMs into the content moderation system. Using normative theories from law and social sciences to critically assess the new technological application, this article seeks to redefine LLMs’ role in content moderation and redirect relevant research in this field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Their operation on socially sedimented corpora reflects and extends human purposes [6,49] while simultaneously introducing new teleological orientations, such as filtering, ranking, and generating content [34].”
From 4E cognition and the coevolution of human–AI interaction (2025) · cites this paper as [34]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“LLM can also self-evaluate their answers by measuring the level of confidence ([109], 3).”
BackgroundHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688 · cited 1×
“It allows us to rethink LLMs’ real potential in content moderation and platform governance, as well as what improvements can be made to further realize its potential.”
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
O’Connor, C., & Joffe, H. (2020) · International Journal of Qualitative Methods · doi:10.1177/1609406919899220 · Cited by 3,871 · cited 1×
“One salient issue, as noted by many researchers in this field, is the intercoder disagreement, referring to the instance that different moderators cannot reach consensus on a case ([87]).”
BackgroundPositivism and the Separation of Law and Morals
Hart, H. L. A. (1958) · Harvard Law Review · doi:10.2307/1338225 · Cited by 1,929
Waldron, J. (1999) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198262138.001.0001 · Cited by 1,649
Çömlekçi, M. F. (2019) · Kırklareli University Institutional Repository (Kırklareli University) · Cited by 1,607
The Brussels Effect: How the European Union Rules the World
Bradford, A. (2020) · doi:10.1093/oso/9780190088583.001.0001 · Cited by 1,349
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245 · cited 1×
“It allows us to rethink LLMs’ real potential in content moderation and platform governance, as well as what improvements can be made to further realize its potential.”
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207 · cited 4×
“And some studies indicated that LLM’s accuracy score is higher than the outsourced human moderators ([46], 2).”
BackgroundMethodInfluentialChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 4×
“And some studies indicated that LLM’s accuracy score is higher than the outsourced human moderators ([46], 2).”
BackgroundMethodInfluential
Research works printed but not linked: 27
- Bonaventura CD et al (2025) From detection to explanation: effective learning strategies for LLMs in online abusive language research. In: Proceedings of the 31st international conference on computational linguistics. Association for computational linguistics, pp 2067–2084. https://aclanthology.org/2025.coling-main.141/
- Bonaventura CD et al (2024) Is explanation all you need? An expert survey on LLM-generated explanations for abusive language detection. In: Tenth Italian conference on computational linguistics (CLiC-it 2024). https://kclpure.kcl.ac.uk/portal/en/publications/is-explanation-all-you-need-an-expert-survey-on-llm-generated-exp
- Yang D et al (2024) On verbalized confidence scores for LLMs. Preprint at arXiv:https://arxiv.org/abs/2412.14737 · arXiv
- Villate-Castillo G et al (2024) A collaborative content moderation framework for toxicity detection based on conformalized estimates of annotation disagreement. Preprint at arXiv:https://arxiv.org/abs/2411.04090v2 · arXiv
- Guerdan L et al (2024) A framework for evaluating LLMs under task indeterminacy. Preprint at arXiv:https://arxiv.org/abs/2411.13760 · arXiv
- Xiong M et al (2024) Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In: The 12th international conference on learning representations. https://openreview.net/forum?id=gjeQKFxFpZ
- Sharma M et al (2023) Towards understanding sycophancy in language models. Preprint at arXiv:https://arxiv.org/abs/2310.13548 · arXiv
- Kroeger N et al (2024) Are large language models post hoc explainers? Preprint at arXiv:https://arxiv.org/abs/2310.05797v3 · arXiv
- Manvi R et al (2024) Large language models are geographically biased. In: Proceedings of the 41st international conference on machine learning. ICML’24. Vienna, Austria
- Wang Y et al (2024) ImpScore: a learnable metric for quantifying the implicitness level of language. Preprint at arXiv:https://arxiv.org/abs/2411.05172v2 · arXiv
- Wang Z et al (2024) MLLM-as-a-judge for image safety without human labeling. Preprint at arXiv:https://arxiv.org/abs/2501.00192 · arXiv
- Bloch-Wehba H (2019) Global platform governance: private power in the shadow of the state. SMU Law Rev 72:27–80
- Boicel A (2024) Using LLMs to moderate content: are they ready for commercial use? https://www.techpolicy.press/using-llms-to-moderate-content-are-they-ready-for-commercial-use/. Accessed 13 Aug 2024
- Brennan-Marquez K (2017) Plausible cause: explanatory standards in the age of powerful machines. Vanderbilt Law Rev 70:1249–1301
- Divakaran DM, Peddinti ST (2024) LLMs for cyber security: new opportunities. Preprint at arXiv:https://arxiv.org/abs/2404.11338 · arXiv
- Elkin-Koren N, Perel M (2023) Speech contestation by design: democratizing speech governance by AI. Fla State Univ Law Rev 50:611–665
- Freeman J (2000) The private role in public governance. New York Univ Law Rev 75:543–675
- Hutchinson AC (1989) Democracy and determinacy: an essay on legal interpretation. Univ Miami Law Rev 43:541–576
- Kaminski ME (2019) Binary governance: lessons from the GDPR’s approach to algorithmic accountability. Southern California Law Rev 92:1529–1616
- Klonick K (2018) The new governors: the people, rules, and processes governing online speech. Harv Law Rev 131:1598–1670
- Meta (2022). How Meta prioritizes content for review. https://transparency.meta.com/policies/improving/prioritizing-content-review/. Accessed 13 Aug 2024
- Meta (2022). How review teams work. https://transparency.meta.com/enforcement/detecting-violations/how-review-teams-work/. Accessed 13 Aug 2024
- Nahmias Y, Perel M (2021) The oversight of content moderation by AI: impact assessments and their limitations. Harv J Legislat 58:145–194
- Schauer F (1985) Easy cases. Southern California Law Rev 58:399–440
- Solum LB (2004) Procedural justice. Southern California Law Rev 78:181–321
- Suzor N et al (2019) What do we mean when we talk about transparency? Toward meaningful transparency in commercial content moderation. Int J Commun 13:1526–1543
- Willner D, Chakrabarti S (2024) Using LLMs for policy-driven content classification. https://www.techpolicy.press/using-llms-for-policy-driven-content-classification/. Accessed 13 Aug 2024
Cited by
Hallucination to truth: a review of fact-checking and factuality evaluation in large language models
Rahman, S. S., Islam, M. A., Alam, M. M., Zeba, M., Rahman, M. A., Chowa, S. S., Raiaan, M. A. K., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11454-w · Cited by 39
“As newer fact-checking techniques, datasets, and benchmarks are published rapidly, it is challenging for researchers and practitioners to keep track of effective solutions [4].”
4E cognition and the coevolution of human–AI interaction
Noller, J. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00595-0 · Cited by 15
Han, R., An, L., Zhou, W., Chen, M., & Li, G. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104808 · Cited by 2
Mittal, P., Sharma, S., Solanki, S., Kumar, L., Haripriya, R., & Beg, R. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01279-z · Cited by 4
Alsayed, A., Elhag, S., & Badri, S. K. (2026) · Information · doi:10.3390/info17010076 · Cited by 1
Metadata
- Authors
- Huang, T.
- Affiliations
- City University of Hong Kong (HK)
- Corresponding author
- Huang, Tao
- ORCID
- Huang, Tao
- Venue
- Artificial Intelligence Review
- Published
- 2025-07-19
- Volume
- 58
- Issue
- 10
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4412962348
- Cited by
- 24
- References
- 80
- Citations by year
- 2026: 19 · 2025: 4
- FWCI
- 24.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by-nc-nd)
- CityU Scholars (submittedVersion, cc-by-nc-nd)
- Funders
- City University of Hong Kong (7200786)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Freedom of Expression and Defamation
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- content moderation, large language models, platform governance, legitimacy, explainable AI, user participation, digital platforms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Content moderation by LLM: from accuracy to legitimacy. https://app.cochat.ai/papers/10.1007/s10462-025-11328-1
- The paper
- Huang, T. (2025). Content moderation by LLM: from accuracy to legitimacy. Artificial Intelligence Review. https://doi.org/10.1007/s10462-025-11328-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.