The DOI matches its DataCite record.
HQP: A Human-Annotated Dataset for Detecting Online Propaganda
Open accessAbstract
Online propaganda poses a severe threat to the integrity of societies. However, existing datasets for detecting online propaganda have a key limitation: they were annotated using weak labels that can be noisy and even incorrect. To address this limitation, our work makes the following contributions: (1) We present HQP: a novel dataset (N = 30,000) for detecting online propaganda with high-quality labels. To the best of our knowledge, HQP is the first large-scale dataset for detecting online propaganda that was created through human annotation. (2) We show empirically that state-of-the-art language models fail in detecting online propaganda when trained with weak labels (AUC: 64.03). In contrast, state-of-the-art language models can accurately detect online propaganda when trained with our high-quality labels (AUC: 92.25), which is an improvement of ~44%. (3) We show that prompt-based learning using a small sample of high-quality labels can still achieve a reasonable performance (AUC: 80.27) while significantly reducing the cost of labeling. (4) We extend HQP to HQP+ to test how well propaganda across different contexts can be detected. Crucially, our work highlights the importance of high-quality labels for sensitive NLP tasks such as propaganda detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Generative AI, specifically LLMs, can be used to spot harmful content on social media.
“generative AI, specifically LLMs, can not only be used to spot harmful content on social media (e.g., Maarouf et al. 2023)”
From Generative AI (2024) · cites this paper as Maarouf et al. 2023
Sections 4.1 and 4.2 describe using pretrained language models to detect propaganda in social-media posts, supporting the claim that AI can spot this type of harmful content. But the detection models are BERT, RoBERTa, and BERTweet; the generative model T5 is used to create prompt templates, not as the final detector. The paper therefore does not establish the claim specifically about generative AI or LLM detectors. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 48
- Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In WWW Companion. · DOI
- Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organiz- ing the news based on their propagandistic content. Information Processing & Management, 56(5):1849- 1864. · DOI
- Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In AAAI. · DOI · Paper page
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
- Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20(1):37-46. · DOI · Paper page
- Giovanni San Da Martino, Stefano Cresci, Alberto Barron-Cedeno, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020. A survey on computational propaganda detection. arXiv. · DOI
- Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. Semeval-2020 task 11: Detection of propaganda techniques in news articles. In Proceed- ings of the Fourteenth Workshop on Semantic Evalu- ation. · DOI
- Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In EMNLP-IJCNLP. · DOI
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL. · DOI · Paper page
- Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Semeval-2021 task 6: Detection of persuasion tech- niques in texts and images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). · DOI
- Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. OpenPrompt: An open-source framework for prompt- learning. In ACL. · DOI
- Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gi- anluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowd- sourcing and characterization of Twitter abusive be- havior. In ICWSM. · DOI
- Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL-IJCNLP. · DOI · Paper page
- Dominique Geissler, Dominik Bär, Nicolas Pröllochs, and Stefan Feuerriegel. 2023. Russian propaganda on social media during the 2022 invasion of ukraine. EPJ Data Science, 12(1). · DOI
- Xiaobo Guo and Soroush Vosoughi. 2022. A large- scale longitudinal multimodal dataset of state-backed information operations on twitter. In ICWSM. · DOI
- Yiqiao Jin, Xiting Wang, Ruichao Yang, Yizhou Sun, Wei Wang, Hao Liao, and Xing Xie. 2022. Towards fine-grained reasoning for fake news detection. In AAAI. · DOI
- Klaus Krippendorff, Yann Mathet, Stéphane Bouvry, and Antoine Widlöcher. 2016. On the reliability of unitizing textual continua: Further developments. Quality & Quantity, 50(6):2347-2364. · DOI
- Valerio La Gatta, Chiyu Wei, Luca Luceri, Francesco Pierri, and Emilio Ferrara. 2023. Retrieving false claims on Twitter during the Russia-Ukraine conflict. In WWW Companion. · DOI
- Junze Li, Mengjie Zhao, Yubo Xie, Antonis Maroniko- lakis, Pearl Pu, and Hinrich Schütze. 2023. This joke is [MASK]: Recognizing humor and offense with prompting. Transfer Learning for Natural Language Processing Workshop.
- Hongzhan Lin, Pengyao Yi, Jing Ma, Haiyun Jiang, Ziyang Luo, Shuming Shi, and Ruifang Liu. 2023. Zero-shot rumor detection with propagation structure via prompt learning. In AAAI. · DOI
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35. · DOI · Paper page
- Yinhan Liu, Myle Ott, Naman Goyal, Du Jingfei, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv.
- Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In ACL. · DOI
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
- Yi-Ju Lu and Cheng-Te Li. 2020. GCAN: Graph-aware co-attention networks for explainable fake news de- tection on social media. In ACL. · DOI · Paper page
- Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. HateXplain: A benchmark dataset for explainable hate speech detection. In AAAI. · DOI
- Pablo Moral, Guillermo Marco, Julio Gonzalo, Jorge Carrillo-de Albornoz, and Iván Gonzalo-Verdugo. 2023. Overview of DIPROMATS 2023: Automatic detection and characterization of propaganda tech- niques in messages from diplomats and authorities of world powers. Procesamiento del Lenguaje Natural, 71(0):397-407.
- Christof Naumzik and Stefan Feuerriegel. 2022. De- tecting false rumors from retweet dynamics on social media. In WWW. · DOI
- Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In EMNLP. Oxford Internet Institute. 2023. Programme on democ- racy & technology. · DOI
- John Pavlopoulos, Leo Laugier, Alexandros Xenos, Jef- frey Sorensen, and Ion Androutsopoulos. 2022. From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer. In ACL. · DOI
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, and Dario Amodei. 2019. Language models are un- supervised multitask learners. Preprint.
- Collin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. JMLR, 21(140):1-67. · DOI
- Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and politi- cal fact-checking. In EMNLP. · DOI
- Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese BERT- networks. In EMNLP-IJCNLP. · DOI · Paper page
- Caitlin M. Rivers and Bryan L. Lewis. 2014. Eth- ical research standards in a world of big data. F1000Research, 3(8). · DOI
- Muhammad Umar Salman, Asif Hanif, Shady Shehata, and Preslav Nakov. 2023. Detecting propaganda techniques in code-switched social media text. In EMNLP.
- Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In NAACL. · DOI · Paper page
- Veronika Solopova, Oana-Iuliana Popescu, Christoph Benzmüller, and Tim Landgraf. 2023. Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts. arXiv. · DOI
- Hyunjin Song, Petro Tolochko, Jakob-Moritz Eberl, Olga Eisele, Esther Greussing, Tobias Heidenreich, Fabienne Lind, Sebastian Galyga, and Hajo G. Boom- gaarden. 2020. In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis. Political Communication, 37(4):550-572. · DOI · Paper page
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for Fact Extraction and VERification. In NAACL. · DOI · Paper page
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. JMLR, 9:2579-2605.
- Prashanth Vijayaraghavan and Soroush Vosoughi. 2022. TWEETSPIN: Fine-grained propaganda detection in social media using multi-view representations. In NAACL. · DOI
- Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, and Songlin Hu. 2021. Towards propagation uncertainty: Edge-enhanced bayesian graph convolutional net- works for rumor detection. In ACL-IJCNLP. · DOI
- Bernard Lewis Welch. 1947. The generalisation of 'stu- dent's' problems when several different population variances are involved. Biometrika, 34(1-2):28-35. · DOI · Paper page
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In EMNLP. · DOI · Paper page
- Rui Xia, Kaizhou Xuan, and Jianfei Yu. 2020. A state- independent and time-evolving network for early ru- mor detection in social media. In EMNLP. · DOI
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.
- Kaimin Zhou, Chang Shu, Binyang Li, and Jey Han Lau. 2019. Early rumour detection. In NAACL. · DOI
Cited by
Sorted by citations
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
“In addition, generative AI, specifically LLMs, can not only be used to spot harmful content on social media (e.g., Maarouf et al. 2023), but it can also create realistic disin-formation (e.g., fake news, propaganda) that is hard to detect by humans (Kreps et al. 2022; Jakesch et al. 2023).”
Self-citationCan LLM-Generated Misinformation Be Detected?Preprint
Chen, C., & Shu, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.13788 · Cited by 224
“: Former President Barack Obama has allegedly been rumored to be involved in planning a conspiracy against the current government during his retirement.”
BackgroundCombating Misinformation in the Age of LLMs: Opportunities and ChallengesPreprint
Chen, C., & Shu, K. (2023) · The AI Magazine · doi:10.48550/arxiv.2311.05656 · Cited by 245
“Previously, there are a large number of works on detecting human-written misinformation including fake news [23, 24, 164, 208, 280, 283, 459, 518, 553, 634], rumor [133, 273, 319, 379], clickbait [74], cherry-picking [25], and propaganda [94, 322, 327].”
BackgroundUsing natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140
“…across different annotators; annotations should be conducted in batches to protect against annotator fatigue, which can influence the quality of annotations 132 ; and the validation sample should be sufficiently large and heterogeneous to capture any idiosyncrasies associated with rare labels.”
Self-citationRussian propaganda on social media during the 2022 invasion of Ukraine
Geissler, D., Bär, D., Pröllochs, N., & Feuerriegel, S. (2023) · EPJ Data Science · doi:10.1140/epjds/s13688-023-00414-5 · Cited by 190
“Here, more research is needed to understand the mechanism behind the pro-Russian propaganda campaign [77], as well as machine learning for detection [78].”
BackgroundSelf-citationChaudhari, D., & Pawar, A. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7040175 · Cited by 11
“LSTM and Bi-LSTM models were used in [6,23,24,26].”
MethodGeissler, D., & Feuerriegel, S. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3686930 · Cited by 8
“Different from this work, previous research aimed at online propaganda was mostly focused on propaganda detection, specifically through natural language processing [9, 31, 44].”
Self-citationTogether we can do it! A roadmap to effectively tackle propaganda-related tasks
Rodríguez-García, R., Centeno, R., & Rodrigo, Á. (2024) · Internet Research · doi:10.1108/intr-05-2024-0785 · Cited by 2
A Trope-Based Computational Approach to Participatory Propaganda Analysis
Niven, T., Li, C., & Lin, Y. (2025) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2025.3592894 · Cited by 1
“The largest strand of prior work attempts a binary judgment of whether or not a text is propaganda [18], [19], [20], [21], [22], [23].”
Fine-grained Narrative Classification in Biased News Articles
Afroz, Z., Vardhan, H., Bhakuni, P., Punia, A., Kumar, R., & Akhtar, M. S. (2025) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2512.03582 · Cited by 1
“Although these datasets advanced the understanding of how propaganda operates linguistically, they remain predominantly in English (Maarouf et al., 2023; Gruppi et al., 2021) and Arabic (Al-Henaki et al., 2025; Alam et al., 2022), overlooking the socio-political and cultural context of countries…”
InfluentialSynthClassify: an LLM-driven framework for generating and classifying persuasive text
Panasyuk, A. (2025) · Defense + Commercial Sensing · doi:10.1117/12.3052408 · Cited by 1
“Other Datasets: Maarouf et al. (2023) [25] address the limitations of existing datasets that rely on noisy, weak labels by creating HQP with high-quality labels through human annotation, consisting of 30,000 posts related to the Russo-Ukrainian war.”
Butkienė, R., Šukys, A., Dambrauskas, E., Žitkus, V., Ablonskis, L., Vaičiukynas, E., Danėnas, P., & Butleris, R. (2026) · Applied Sciences · doi:10.3390/app16126251
“[46] showed that prompt-based learning can reduce annotation costs while maintaining high label quality.”
Political Persuasion and Endorsement in Large Language ModelsPreprint
Antelmi, A., Galdeman, A., Cava, L. L., Pera, A., & Martino, G. D. S. (2026) · arXiv.org · doi:10.48550/arxiv.2606.05961
“The first dataset consists of 29,596 tweets collected during the beginning of the Ukraine–Russia conflict [28], each annotated with one fine-grained persuasion technique or as containing no persuasion techniques.”
Metadata
- Authors
- Maarouf, A., Bär, D., Geissler, D., & Feuerriegel, S.
- Affiliations
- Munich Center for Machine Learning (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 6064-6089
- arXiv
- 2304.14931
- OpenAlex
- W4402670741
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 3
- FWCI
- 5.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- human-annotated dataset, weak labels, label noise, label quality, prompt learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
