Auditing Algorithmic Bias on Twitter
Open accessAbstract
Digital media platforms are reshaping our habits, how we access information, and how we interact with others. As a result, algorithms used by platforms, for example, to recommend content, play an increasingly important role in our access to information. Due to practical difficulties of accessing how platforms present content to their users, relatively little is known about how recommendation algorithms affect the information people receive. In this paper we implement a sock-puppet audit, a computational framework to audit black-box social media systems so as to quantify the impact of algorithmic curation on the information people see. We evaluate this framework by conducting a study on Twitter. We demonstrate that Twitter’s timeline curation algorithms skew the popularity and novelty of content people see and increase the inequality of their exposure to friends’ tweets. Our work provides evidence that algorithmic curation of content systematically distorts the information people see.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even beyond the friend network, exposure inequality remains at a similar level to that reported for friends' tweets, indicating that the platform's algorithm amplifies certain accounts both within and outside of users' direct networks.
“This suggests that even beyond the friend network, exposure inequality remains at a similar level, indicating that the platform's algorithm amplifies certain accounts both within and outside of users' direct networks.”
From Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X During the 2024 U.S. Presidential Election (2025) · cites this paper as [6]
Section 4.3.1 reports that exposure to friends’ tweets is more unequal in the personalized timeline, and Section 5 attributes a disproportionate share of that timeline to a few friends. Neither section reports exposure inequality beyond the friend network or compares its level with inequality for friends’ tweets. Page numbers are unavailable in our copy. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPrevious sock-puppet audits on X are often constrained by small sample sizes of fewer than 10 accounts and restricted tweet collection in quantity and frequency, limiting the generalization and robustness of their findings.
“Furthermore, previous sock-puppet audits on X are often constrained by small sample sizes (fewer than 10 accounts) and restricted tweet collection in terms of both quantity and frequency [4,6,9], limiting the generalization and robustness of their findings.”
From Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X During the 2024 U.S. Presidential Election (2025) · cites this paper as [6]
Section 3.2 (Audit Bots; page unavailable) reports that this audit used eight accounts because of scaling issues and collected at least 15 tweets per session, six times a day. Section 3.3 (Timeline Scraping; page unavailable) says the observed tweets represented less than 10% of content produced by the accounts the bots followed. These passages support the description of this audit’s scale and collection limits, but do not establish that previous audits were often similarly constrained or state that those limits reduced the generalizability and robustness of their findings. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPrevious research has shown that popularity biases can lead to a skew in the visibility of tweets when comparing personalized feeds with reverse-chronological ones, and that users are disproportionately exposed to friends' tweets.
“Previous research has shown that popularity biases can lead to a skew in the visibility of tweets when comparing personalized feeds with reverse-chronological ones, and that users are disproportionately exposed to friends' tweets [6,7].”
From Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X During the 2024 U.S. Presidential Election (2025) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,219 · cited 1× · see citing works
“Moreover, existing cognitive biases, including social influence [28] and position bias [20], coupled with algorithmic recommendations can amplify online trends, creating “irrational herding” and distorting the perceptions of the underlying value of content [1, 2, 11].”
BackgroundFilter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992 · see citing works
How algorithmic popularity bias hinders or promotes quality
Ciampaglia, G. L., Nematzadeh, A., Menczer, F., & Flammini, A. (2018) · Scientific Reports · doi:10.1038/s41598-018-34203-2 · Cited by 204 · see citing works
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · see citing works
Leveraging position bias to improve peer recommendation
Lerman, K., & Hogg, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098914 · Cited by 96 · see citing works
Self-citationThe Effect of Recommendations on Network Structure
Su, J., Sharma, A., & Goel, S. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883040 · Cited by 78 · see citing works
Flagged first, then by citations
Research works printed but not linked: 24
- Andrés Abeliuk, Gerardo Berbeglia, Manuel Cebrian, and Pascal Van Hentenryck. 2015. The benefits of social influence in optimized cultural markets. PloS one 10, 4 (2015), e0121934.
- Andrés Abeliuk, Gerardo Berbeglia, Pascal Van Hentenryck, Tad Hogg, and Kristina Lerman. 2017. Taming the unpredictability of cultural markets with social influence. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 745-754.
- Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (2018), 54-61.
- Eytan Bakshy, Solomon Messing, and Lada A Adamic. 2015. Exposure to ideologi- cally diverse news and opinion on Facebook. Science 348, 6239 (2015), 1130-1132.
- Jack Bandy and Nicholas Diakopoulos. 2020. Auditing news curation systems: A case study examining algorithmic and editorial logic in apple news. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 36-47.
- Michael A Beam. 2014. Automating the news: How personalized news recom- mender system design choices impact news reception. Communication Research 41, 8 (2014), 1019-1041.
- Òscar Celma and Pedro Cano. 2008. From hits to niches? or how popular artists can bias music recommendation and discovery. In Proceedings of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition. 1-8.
- Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2018. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 224-232.
- Wen Chen, Diogo Pacheco, Kai-Cheng Yang, and Filippo Menczer. 2020. Neutral Bots Reveal Political Bias on Social Media. arXiv preprint arXiv:2005.08141 (2020). · arXiv
- Sunshine Chong and Andrés Abeliuk. 2019. Quantifying the Effects of Recom- mendation Systems. In 2019 IEEE International Conference on Big Data (Big Data). IEEE, 3008-3015.
- Santo Fortunato, Alessandro Flammini, Filippo Menczer, and Alessandro Vespig- nani. 2006. Topical interests and the mitigation of search engine bias. Proceedings of the national academy of sciences 103, 34 (2006), 12684-12689.
- Eduardo Hargreaves, Claudio Agosti, Daniel Menasché, Giovanni Neglia, Alexan- dre Reiffers-Masson, and Eitan Altman. 2018. Biases in the facebook news feed: a case study on the italian elections. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE, 806-812.
- Nathan O Hodas and Kristina Lerman. 2014. The simple rules of social contagion. Scientific reports 4 (2014), 4343.
- Dietmar Jannach, Lukas Lerche, Iman Kamehkhosh, and Michael Jugovac. 2015. What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25, 5 (2015), 427-491.
- Solomon Messing and Sean J Westwood. 2014. Selective exposure in the age of social media: Endorsements trump partisan source affiliation when selecting news online. Communication research 41, 8 (2014), 1042-1063.
- Dimitar Nikolov, Mounia Lalmas, Alessandro Flammini, and Filippo Menczer. 2019. Quantifying biases in online information exposure. Journal of the Associa- tion for Information Science and Technology 70, 3 (2019), 218-229.
- Yoon-Joo Park and Alexander Tuzhilin. 2008. The long tail of recommender systems and how to leverage it. In Proceedings of the 2008 ACM conference on Recommender systems. 11-18.
- Matjaž Perc. 2014. The Matthew effect in empirical data. Journal of The Royal Society Interface 11, 98 (2014), 20140378.
- Ashwin Rao, Fred Morstatter, Minda Hu, Emily Chen, Keith Burghardt, Emilio Ferrara, and Kristina Lerman. 2020. Political Partisanship and Anti-Science Attitudes in Online Discussions about Covid-19. arXiv preprint arXiv:2011.08498 (2020). WebSci '21, June 21-25, 2021, Virtual Event, United Kingdom · arXiv
- Ronald E Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018. Auditing partisan audience bias within google search. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-22.
- Manuel Gomez Rodriguez, Krishna Gummadi, and Bernhard Schoelkopf. 2014. Quantifying information overload in social media and its impact on social conta- gions. In Eighth International AAAI Conference on Weblogs and Social Media.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014), 4349-4357.
- Siqi Wu, Marian-Andrei Rizoiu, and Lexing Xie. 2020. Variation across Scales: Measurement Fidelity under Twitter Data Sampling. In Proceedings of the Inter- national AAAI Conference on Web and Social Media, Vol. 14. 715-725.
- JungAe Yang. 2016. Effects of popularity-based news recommendations ("most- viewed") on users' exposure to online news. Media Psychology 19, 2 (2016), 243-271.
Cited by
Sorted by citations
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152 · see citing works
“Reflecting that convergence, previous research has developed procedures to audit AI systems for transparency and explainability (Cobbe et al., 2021; Mittelstadt, 2016), bias and fairness (Bartley et al., 2021; Morina et al., 2019) and accountability (Busuioc, 2021; Metcalf et al., 2021).”
MethodPedreschi, D., Pappalardo, L., Ferragina, E., Baeza-Yates, R., Barabási, A.-L., Dignum, F., Dignum, V., Eliassi-Rad, T., Giannotti, F., Kertész, J., et al. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2024.104244 · Cited by 102 · see citing works
Wojcieszak, M., Casas, A., Yu, X., Nagler, J., & Tucker, J. A. (2022) · Science Advances · doi:10.1126/sciadv.abn9418 · Cited by 90 · see citing works
Ye, J., Luceri, L., & Ferrara, E. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732159 · Cited by 16 · see citing works
“Previous studies consistently show that X ’s algorithm amplifies political biases and prioritizes high-engagement content, including emotionally charged, toxic, and low-credibility information [4, 6, 8–10 , 20].”
InfluentialSelf-citationWang, S., Huang, S., Zhou, A., & Metaxa, D. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687046 · Cited by 11 · see citing works
- Authors
- Bartley, N., Abeliuk, A., Ferrara, E., & Lerman, K.
- Affiliations
- University of Southern California (US)
- University of Chile, Chile
- Venue
- 13th ACM Web Science Conference 2021
- Publisher
- ACM
- Published
- 2021-06-21
- Pages
- 65-73
- OpenAlex
- W3173463346
- Cited by
- 62
- References
- 6
- Citations by year
- 2026: 10 · 2025: 10 · 2024: 14 · 2023: 13 · 2022: 15
- FWCI
- 9.49
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Web Science Conference (WebSci) (publishedVersion)
- Funders
- DARPA (W911NF-18-C-0011)
- AFOSR (FA9550-20-1-0224)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- algorithmic bias, Twitter, recommendation algorithms, information exposure, content popularity, novelty, algorithmic auditing, information inequality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
