Crowdsourcing the Annotation of Rumourous Conversations in Social Media
Abstract
Social media are frequently rife with rumours, and the study of rumour conversational aspects can provide valuable knowledge about how rumours evolve over time and are discussed by others who support or deny them. In this work, we present a new annotation scheme for capturing rumour-bearing conversational threads, as well as the crowdsourcing methodology used to create high quality, human annotated datasets of rumourous conversations from social media. The rumour annotation scheme is validated through comparison between crowdsourced and reference annotations. We also found that only a third of the tweets in rumourous conversations contribute towards determining the veracity of rumours, which reinforces the need for developing methods to extract the relevant pieces of information automatically.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In closed scenarios, a simple approach is to collect as many posts as possible for an event, then filter tweets associated with rumours.
“In closed scenarios where there is a need to make sure that rumours that emerge during an event or news story are collected, the simple approach is to collect as many posts as possible for those events.”
From Detection and Resolution of Rumours in Social Media (2019) · cites this paper as Zubiaga et al. 2015
With bottom-up sampling, manual annotation is needed to distinguish rumours from non-rumours.
“However, when using a bottom-up sampling strategy, manual annotation work is needed to identify what constitutes a rumour and what a non-rumour (Zubiaga et al. 2015).”
From Detection and Resolution of Rumours in Social Media (2019) · cites this paper as Zubiaga et al. 2015
Cites
What is Twitter, a social network or a news media?
Kwak, H., Lee, C., Park, H., & Moon, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772751 · Cited by 6,652 · see citing works
Misinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012) · Psychological Science in the Public Interest · doi:10.1177/1529100612451018 · Cited by 2,920 · see citing works
Processing Social Media Messages in Mass Emergency: A Survey
Imran, M., Castillo, C., Diaz, F., & Vieweg, S. (2015) · ACM Computing Surveys · doi:10.1145/2771588 · Cited by 804 · see citing works
(2011) · Cited by 724 · see citing works
Lee, A. M., Allport, G. W., & Postman, L. J. (1948) · American Sociological Review · doi:10.2307/2086591 · Cited by 702 · see citing works
(2014) · iConference 2014 Proceedings · doi:10.9776/14308 · Cited by 423 · see citing works
Rumor and Gossip, the Social Psychology of Hearsay
Klapp, O. E., Rosnow, R. L., & Fine, G. A. (1977) · The Canadian Journal of Sociology · doi:10.2307/3340300 · Cited by 367 · see citing works
Predicting information credibility in time-sensitive social media
Castillo, C., Mendoza, M., & Poblete, B. (2013) · Internet Research · doi:10.1108/intr-05-2012-0095 · Cited by 339 · see citing works
FactBank: a corpus annotated with event factuality
Sauri, R. S., & Pustejovsky, J. D. (2009) · Language Resources and Evaluation · doi:10.1007/s10579-009-9089-9 · Cited by 292 · see citing works
Annotating named entities in Twitter data with crowdsourcing
Finin, T., Murnane, W., Karandikar, A., Keller, N., Martineau, J., & Dredze, M. H. (2010) · doi:10.13016/m21j97c2p · Cited by 282 · see citing works
Reading the riots on Twitter: methodological innovation for the analysis of big data
Procter, R., Vis, F., & Voß, A. (2013) · International Journal of Social Research Methodology · doi:10.1080/13645579.2013.774172 · Cited by 271 · see citing works
Self-citationA Study of a Rumor: Its Origin and Spread
Festinger, L., Cartwright, D., Barber, K., Fleischl, J., Gottsdanker, J., Keysen, A., & Leavitt, G. (1948) · Human Relations · doi:10.1177/001872674800100405 · Cited by 179 · see citing works
Reading the riots: what were the police doing on Twitter?
Procter, R., Crump, J., Karstedt, S., Voß, A., & Cantijoch, M. (2013) · Policing & Society · doi:10.1080/10439463.2013.780223 · Cited by 135 · see citing works
Self-citationTweet, but verify: epistemic study of information verification on Twitter
Zubiaga, A., & Ji, H. (2014) · Social Network Analysis and Mining · doi:10.1007/s13278-014-0163-y · Cited by 125 · see citing works
Self-citationCheng, J., Teevan, J. B., Iqbal, S. T., & Bernstein, M. S. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702146 · Cited by 122 · see citing works
Rumor and Gossip: The Social Psychology of Hearsay.
Vuchinich, S., Rosnow, R. L., & Fine, G. A. (1977) · Social Forces · doi:10.2307/2577775 · Cited by 78 · see citing works
Towards Detecting Rumours in Social Media
Zubiaga, A., Liakata, M., Procter, R., Bontcheva, K., & Tolmie, P. (2015) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.1504.04712 · Cited by 72 · see citing works
Self-citationCoordinating human and machine intelligence to classify microblog communications in crises
Imran, M., Castillo, C., Lucas, J. R., Meier, P. P., & Rogstadius, J. (2014) · International Conference on Information Systems for Crisis Response and Management · Cited by 66 · see citing works
Rumor and Gossip: The Social Psychology of Hearsay
Rosnow, R. L., & Fine, G. A. (1976) · TUScholarShare (Temple University) · Cited by 40 · see citing works
What is a Question? Crowdsourcing Tweet Categorization
Paul, S. A., Hong, L., & Ed, H. (2011) · Cited by 32 · see citing works
Enabling social media research through citizen social science
Procter, R., Alex Voss, L., Housley, W., Edwards, A., Williams, M. L., Burnap, P., RANA, O. F., & Sloan, L. (2013) · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Zubiaga, A., Liakata, M., Procter, R., Wong Sak Hoi, G., & Tolmie, P. (2016) · PLOS ONE · doi:10.1371/journal.pone.0150989 · Cited by 722 · see citing works
Self-citationDetection and Resolution of Rumours in Social Media
Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., & Procter, R. (2019) · ACM Computing Surveys · doi:10.1145/3161603 · Cited by 608 · see citing works
Self-citationExploiting Context for Rumour Detection in Social Media
Zubiaga, A., Liakata, M., & Procter, R. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67217-5_8 · Cited by 350 · see citing works
Self-citationSemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours
Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Wong Sak Hoi, G., & Zubiaga, A. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2006 · Cited by 323 · see citing works
Self-citationLearning Reporting Dynamics during Breaking News for Rumour Detection in Social MediaPreprint
Zubiaga, A., Liakata, M., & Procter, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07363 · Cited by 162 · see citing works
Self-citationDiscourse-aware rumour stance classification in social media using sequential classifiers
Zubiaga, A., Kochkina, E., Liakata, M., Procter, R., Łukasik, M., Bontcheva, K., Cohn, T., & Augenstein, I. (2017) · Information Processing & Management · doi:10.1016/j.ipm.2017.11.009 · Cited by 149 · see citing works
Self-citationTree LSTMs with Convolution Units to Predict Stance and Rumor Veracity in Social Media Conversations
Kumar, S., & Carley, K. M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1498 · Cited by 120 · see citing works
Fernández, M., & Alani, H. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3188730 · Cited by 111 · see citing works
Unsupervised rumor detection based on users’ behaviors using neural networks
Chen, W., Zhang, Y., Yeo, C. K., Lau, C. T., & Lee, B. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.10.014 · Cited by 100 · see citing works
Ciampaglia, G. L. (2017) · Journal of Computational Social Science · doi:10.1007/s42001-017-0005-6 · Cited by 84 · see citing works
A review on rumour prediction and veracity assessment in online social network
Varshney, D., & Vishwakarma, D. K. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114208 · Cited by 82 · see citing works
Nowhere to Hide: Online Rumor Detection Based on Retweeting Graph Neural Networks
Liu, B., Sun, X., Meng, Q., Yang, X., Lee, Y., Cao, J., Luo, J., & Lee, R. K.-W. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3161697 · Cited by 49 · see citing works
Computational Rumor Detection Without Non-Rumor: A One-Class Classification Approach
Fard, A. E., Mohammadi, M., Chen, Y., & de Walle, B. V. (2019) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2019.2931186 · Cited by 49 · see citing works
Review of stance detection for rumor verification in social media
Alsaif, H. F., & Al-Dossari, H. Z. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105801 · Cited by 34 · see citing works
Tracing the fake news propagation path using social network analysis
Sivasankari, S., & Vadivu, G. (2021) · Soft Computing · doi:10.1007/s00500-021-06043-2 · Cited by 33 · see citing works
Eismann, K. (2021) · Journal of Business Economics · doi:10.1007/s11573-020-01022-9 · Cited by 34 · see citing works
Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing
Lim, S., Jatowt, A., Färber, M., & Yoshikawa, M. (2020) · Language Resources and Evaluation · Cited by 32 · see citing works
Wang, X., Chao, F., & Yu, G. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.770111 · Cited by 30 · see citing works
Unsupervised Cross-Domain Rumor Detection with Contrastive Learning and Cross-Attention
Ran, H., & Jia, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26584 · Cited by 29 · see citing works
PHEME : computing veracity : the fourth challenge of big social data
Derczynski, L., Bontcheva, K., Łukasik, M., Declerck, T., Arno Scharl, A., Georgiev, G., Osenova, P., Lobo, T. P., Anna, K., Stewart, R. J., et al. (2014) · Warwick Research Archive Portal (University of Warwick) · doi:10.82444/warw.33697567 · Cited by 29 · see citing works
Self-citationMicroblog Analysis as a Program of Work
Tolmie, P., Procter, R., Rouncefield, M., Liakata, M., & Zubiaga, A. (2018) · ACM Transactions on Social Computing · doi:10.1145/3162956 · Cited by 14 · see citing works
Self-citationLorentzen, D. G., & Nolin, J. M. (2015) · Social Science Computer Review · doi:10.1177/0894439315607018 · Cited by 25 · see citing works
An Analysis of Event-Agnostic Features for Rumour Classification in Twitter
Toloşi, L., Tagarev, A., & Georgiev, G. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i2.14844 · Cited by 20 · see citing works
The classification of rumour standpoints in online social network based on combinatorial classifiers
Ma, J., & Luo, Y. (2019) · Journal of Information Science · doi:10.1177/0165551519828619 · Cited by 19 · see citing works
Understanding Characteristics of Biased Sentences in News Articles.
Lim, S., Jatowt, A., & Yoshikawa, M. (2018) · Cited by 18 · see citing works
Deep learning-based credibility conversation detection approaches from social network
Fadhli, I., Hlaoua, L., & Omri, M. N. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01066-z · Cited by 15 · see citing works
C-STANCE: A Large Dataset for Chinese Zero-Shot Stance Detection
Zhao, C., Li, Y., & Caragea, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.747 · Cited by 14 · see citing works
Regroup Median Loss for Combating Label Noise
Li, F., Li, K., Tian, J., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29250 · Cited by 14 · see citing works
Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation
Gartland, A., Bate, A., Painter, J. L., Casperson, T. A., & Powell, G. E. (2020) · Drug Safety · doi:10.1007/s40264-020-01028-w · Cited by 13 · see citing works
MTLTS: A Multi-Task Framework To Obtain Trustworthy Summaries From Crisis-Related Microblogs
Mukherjee, R., Vishnu, U., Peruri, H. C., Bhattacharya, S., Rudra, K., Goyal, P., & Ganguly, N. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498536 · Cited by 13 · see citing works
A Stochastic Attention CNN Model for Rumor Stance Classification
Bai, N., Wang, Z., & Meng, F. (2020) · IEEE Access · doi:10.1109/access.2020.2990770 · Cited by 12 · see citing works
Birds of a feather check together: Leveraging homophily for sequential rumour detection
Lathiya, S., Dhobi, J. S., Zubiaga, A., Liakata, M., & Procter, R. (2020) · Online Social Networks and Media · doi:10.1016/j.osnem.2020.100097 · Cited by 12 · see citing works
Self-citationSynews: a synergy-based rumor verification system
Sarfraz, A., Ahmad, A., Zeshan, F., & Mirza, H. T. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01214-z · Cited by 11 · see citing works
Ontological Modelling of Rumors
Declerck, T., Osenova, P., Georgiev, G., & Lendvai, P. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-32942-0_1 · Cited by 10 · see citing works
Explainable Rumor Detection using Inter and Intra-feature Attention NetworksPreprint
Chen, M., Wang, N., & Subbalakshmi, K. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11057 · Cited by 10 · see citing works
Soprano, M., Roitero, K., Bona, F. B. D., & Mizzaro, S. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3502182 · Cited by 7 · see citing works
Sharma, A. K., Rudra, K., & Ganguly, N. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3358020 · Cited by 7 · see citing works
Zhang, C., Wang, L., Lin, Y., & Wang, X. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3468017 · Cited by 6 · see citing works
Unsupervised contrastive domain adaptive rumor detection with test-time classifier adjustment
Ran, H., Zhang, D., Li, X., Ma, H., Jia, C., & Feng, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104341 · Cited by 6 · see citing works
An Implicit Crowdsourcing Approach to Rumor Identification in Online Social Networks
Osho, A., Waters, C., & Amariucai, G. T. (2020) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam49781.2020.9381339 · Cited by 6 · see citing works
Graph Enhanced BERT for Stance-Aware Rumor Verification on Social Media
Ye, K., Piao, Y., Zhao, K., & Cui, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86383-8_34 · Cited by 6 · see citing works
Rumour As an Anomaly: Rumour Detection with One-Class Classification
Fard, A. E., Mohammadi, M., Cunningham, S. W., & de Walle, B. V. (2019) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice.2019.8792644 · Cited by 6 · see citing works
Analysis of Research on Online Rumors
Pal, A., & Chua, A. Y. K. (2019) · International Conference on Information Management · doi:10.1109/infoman.2019.8714707 · Cited by 5 · see citing works
Hajarian, M., Dı́az, P., & Aedo, I. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2427989 · Cited by 4 · see citing works
Analysis on Potential Use of Crowdsourcing in Different Domain Using Metasynthesis
Kasturi, N., Totad, S. G., & Ghosh, G. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-4193-1_73 · Cited by 4 · see citing works
CREDEEP: Deep Learning-based approaches to detect credibility in Twitter conversations
Fadhli, I., Hlaoua, L., & Omri, M. N. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.10.384 · Cited by 4 · see citing works
Multi-stage dynamic disinformation detection with graph entropy guidance
Hao, X., Liu, B., Yang, X., Sun, X., Meng, Q., & Cao, J. (2024) · World Wide Web · doi:10.1007/s11280-024-01243-w · Cited by 4 · see citing works
Detection and Resolution of Rumors and Misinformation with NLP
Derczynski, L., & Zubiaga, A. (2020) · doi:10.18653/v1/2020.coling-tutorials.4 · Cited by 3 · see citing works
Self-citationMultilevel Feature Fusion-Based GCN for Rumor Detection with Topic Relevance Mining
Chen, S., Li, M., & Yang, W. (2023) · Advances in Multimedia · doi:10.1155/2023/5975153 · Cited by 3 · see citing works
Socio-Technical Mitigation Effort to Combat Cyber Propaganda: A Systematic Literature Mapping
Maseri, A. N., Norman, A. A., Eke, C. I., Ahmad, A., & Molok, N. N. A. (2020) · IEEE Access · doi:10.1109/access.2020.2994658 · Cited by 2 · see citing works
FTR-18: Collecting rumours on football transfer newsPreprint
Caled, D., & Silva, M. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00778 · Cited by 2 · see citing works
Danish Stance Classification and Rumour ResolutionPreprint
Lillie, A. E., & Middelboe, E. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01304 · Cited by 2 · see citing works
Liao, J., Cherng, F., Singh, M., & Wang, H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713426 · Cited by 1 · see citing works
Xu, M., Zhang, H., Chang, H., Si, N., Qu, D., & Hao, C. (2024) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait63902.2024.11021876 · Cited by 1 · see citing works
PHEME D2.4: Qualitative Analysis of Rumours, Sources, and Diffusers Across Media and Languages
Zubiaga, A., Tolmie, P., Liakata, M., & Procter, R. (2015) · Cited by 1 · see citing works
Self-citationDesigning prototype for accessing Veracity using social sensors - big data perspective
Parvathi, A. V., & Karthika, S. (2017) · International Conference on Advances in Electrical, Electronics, Information, Communication and Bio-Informatics · doi:10.1109/aeeicb.2017.7972411 · Cited by 1 · see citing works
Following Tweets Around : Informetric methodology for the Twittersphere
Lorentzen, D. G. (2016) · Cited by 1 · see citing works
A Weakly Supervised Approach for Classifying Stance in Twitter Replies
Kumar, S., Cox, R. A. V., Babcock, M., & Carley, K. M. (2021) · Digital Eprints Services at ISB (DESI) (Indian School of Business) · doi:10.48550/arxiv.2103.07098 · Cited by 2 · see citing works
Towards better prediction and content detection through online social media mining
Chen, W. (2018) · doi:10.32657/10356/75925 · Cited by 1 · see citing works
Ontology-based rumor identification system using social network dataPreprint
Sarfraz, A., Ahmad, A., Zeshan, F., & Ahmed, S. (2022) · Research Square · doi:10.21203/rs.3.rs-1583922/v1 · Cited by 1 · see citing works
#BiggBoss - Twitter Driven Temporal Peak Recognition in Long Run Events
Parvathi, A. V., Karthika, S., & Bose, S. (2017) · International Conference on Advanced Computing · doi:10.1109/icoac.2017.8441477
Prevention of Ethnic Violence and Riots Using Supervised Rumor Detection
Das, R. K., Panda, M., Apoorva, A., & Mishra, G. K. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-8676-3_48
Determining Event Outcomes: The Case of #fail
Murugan, S., Chinnappa, D., & Blanco, E. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.359
Analysis of Structural Features in Rumor Conversations Detection in Twitter
Lotfi, S., Mirzarezaee, M., Hosseinzadeh, M., & Seydi, V. (2021) · Signal and Data Processing · doi:10.52547/jsdp.18.3.45
Social context in political stance detection: Impact and extrapolation
Cox, R. A. V., Williams, E., & Carley, K. M. (2025) · PLoS ONE · doi:10.1371/journal.pone.0324697
- Authors
- Zubiaga, A., Liakata, M., Procter, R., Bontcheva, K., & Tolmie, P.
- Affiliations
- University of Warwick (GB)
- University of Sheffield (GB)
- Venue
- Proceedings of the 24th International Conference on World Wide Web
- Publisher
- ACM
- Published
- 2015-05-18
- Pages
- 347-353
- OpenAlex
- W1906632846
- Cited by
- 67
- References
- 23
- Citations by year
- 2025: 3 · 2024: 6 · 2023: 6 · 2022: 5 · 2021: 7 · 2020: 10 · 2019: 9 · 2018: 7 · 2017: 7 · 2016: 4 · 2015: 2 · 2014: 1
- FWCI
- 24.30
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International World Wide Web Conference (WWW) (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- Funders
- European Commission FP7 (611233)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Complex Network Analysis Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Keywords
- rumor detection, annotation scheme, crowdsourcing, social media, tweets, annotated dataset, information extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
