Bot, or not? Comparing three methods for detecting social bots in five political discourses
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Social bots – partially or fully automated accounts on social media platforms – have not only been widely discussed, but have also entered political, media and research agendas. However, bot detection is not an exact science. Quantitative estimates of bot prevalence vary considerably and comparative research is rare. We show that findings on the prevalence and activity of bots on Twitter depend strongly on the methods used to identify automated accounts. We search for bots in political discourses on Twitter, using three different bot detection methods: Botometer, Tweetbotornot and “heavy automation”. We drew a sample of 122,884 unique user Twitter accounts that had produced 263,821 tweets contributing to five political discourses in five Western democracies. While all three bot detection methods classified accounts as bots in all our cases, the comparison shows that the three approaches produce very different results. We discuss why neither manual validation nor triangulation resolves the basic problems, and conclude that social scientists studying the influence of social bots on (political) communication and discourse dynamics should be careful with easy-to-use methods, and consider interdisciplinary research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1×
“Vosoughi et al. (2018: 1146) investigated the diffusion of true and false news on Twitter, fi nding that humans and bots spread false information, which diffused ‘ farther, faster, deeper, and more broadly ’ in social networks than true information did.”
BackgroundWeaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149
Social bots distort the 2016 U.S. Presidential election online discussion
Bessi, A., & Ferrara, E. (2016) · First Monday · doi:10.5210/fm.v21i11.7090 · Cited by 993 · cited 11×
“Bessi and Ferrara (2016) concluded that bots were responsible for about one-fifth of the Twitter conversations about the US presidential election. They further estimated that bots accounted for about 15% of the total Twitter population involved in these discussions (Ferrara, 2018). Wojcik et al. (2018) argued that bot…”
InfluentialOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 3×
“Botometer has to date been used in various studies: Botometer ’ s developers estimated that the bot population on Twitter ranges between 9% and 15% (Varol et al., 2017).”
BackgroundMethodInfluentialBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909 · cited 13×
“The second tool in our study, Botometer (Davis et al., 2016), is a particularly well-established bot detection tool that many in fl uential studies and organisations, for example the PEW Research Center, use (Wojcik et al., 2018).”
BackgroundMethodInfluentialDetecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg?
Chu, Z., Gianvecchio, S., Wang, H., & Jajodia, S. (2012) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2012.75 · Cited by 634 · cited 1×
“By analysing a random sample of Twitter users, Chu et al. (2012) estimate that the Twitter population comprises human, cyborg and bot accounts in a 5:4:1 ratio.”
BackgroundDisinformation and social bot operations in the run up to the 2017 French presidential election
Ferrara, E. (2017) · First Monday · doi:10.5210/fm.v22i8.8005 · Cited by 524
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · cited 4×
“The developers, computer scientists at the University of Indiana, de fi ne social bots as ‘ automated or semi-automated accounts designed to impersonate humans ’ (Yang et al., 2019: 48).”
BackgroundInfluentialBots, #Strongerin, and #Brexit: Computational Propaganda During the UK-EU Referendum
Howard, P. N., & Kollanyi, B. (2016) · doi:10.2139/ssrn.2798311 · Cited by 392
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · cited 2×
“Studies have already shown that human coders are by no means better able to identify automated accounts (Cresci et al., 2017a, 2017b), at least in manual content analysis, while sophisticated digital forensics is a different story.”
Background
Cited by
Gil de Zúñiga, H., Goyanes, M., & Durotoye, T. (2024) · Political Communication · doi:10.1080/10584609.2023.2290497 · Cited by 198
“There are cases of Twitter bot indicators - Botometer and Tweetbotornot - performing tasks like detecting automated accounts that collate tweets used to promote political discourses (Martini et al., 2021).”
BackgroundBotometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159
Blanco-Alfonso, I., Rodríguez-Fernández, L., & Arce-García, S. (2022) · Revista de Comunicación · doi:10.26441/rc21.2-2022-a2 · Cited by 38
Casas-Mas, B., Gómez González, D., & Simón de Blas, C. (2026) · Redes. Revista hispana para el análisis de redes sociales · doi:10.5565/rev/redes.1135
Metadata
- Authors
- Martini, F., Samula, P., Keller, T. R., & Klinger, U.
- Venue
- Big Data & Society
- Publisher
- SAGE Publications
- Published
- 2021-07
- Volume
- 8
- Issue
- 2
- ISSN
- 2053-9517 (ISSN-L)
- OpenAlex
- W3194561115
- Cited by
- 91
- References
- 21
- FWCI
- 22.92
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Bot, or not? Comparing three methods for detecting social bots in five political discourses. https://app.cochat.ai/papers/10.1177/20539517211033566
- The paper
- Martini, F., Samula, P., Keller, T. R., & Klinger, U. (2021). Bot, or not? Comparing three methods for detecting social bots in five political discourses. Big Data & Society. https://doi.org/10.1177/20539517211033566
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.