Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose
Open accessAbstract
Twitter is a social media giant famous for the exchange of short, 140-character messages called "tweets". In the scientific community, the microblogging site is known for openness in sharing its data. It provides a glance into its millions of users and billions of tweets through a "Streaming API" which provides a sample of all tweets matching some parameters preset by the API user. The API service has been used by many researchers, companies, and governmental institutions that want to extract knowledge in accordance with a diverse array of questions pertaining to social media. The essential drawback of the Twitter API is the lack of documentation concerning what and how much data users get. This leads researchers to question whether the sampled data is a valid representation of the overall activity on Twitter. In this work we embark on answering this question by comparing data collected using Twitter's sampled API service with data collected using the full, albeit costly, Firehose stream that includes every single published tweet. We compare both datasets using common statistical metrics as well as metrics that allow us to compare topics, networks, and locations of tweets. The results of our work will help researchers and practitioners understand the implications of using the Streaming API.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some platforms provide APIs to access data, but they set limitations on quantity of data collected within a timeframe, and provide query languages of limited expressiveness.
“Some platforms provide Application Programmer Interfaces (APIs) to access data, but they set limitations on the quantity of data that can be collected within a given timeframe, and provide query languages of limited expressiveness”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Morstatter et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 35
- Agresti, A. 2010. Analysis of Ordinal Categorical Data, volume 656. Hoboken, New Jersey: Wiley.
- Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirich- let allocation. The Journal of Machine Learning Research 3:993-1022.
- Borgatti, S. P.; Carley, K. M.; and Krackhardt, D. 2006. On the robustness of centrality measures under conditions of imperfect data. Social Networks 28(2):124-136.
- Casella, G., and Berger, R. L. 2001. Statistical Inference. Belmont, CA: Duxbury Press.
- Chae, J.; Thom, D.; Bosch, H.; Jang, Y.; Maciejewski, R.; Ebert, D. S.; and Ertl, T. 2012. Spatiotemporal Social Media Analytics for Abnormal Event Detection and Examination using Seasonal-Trend Decomposition. In Proceedings of the IEEE Conference on Visual Analytics Science and Technol- ogy. Brighton, UK: IEEE Conference on Visual Analytics Science and Technology.
- Cheng, Z.; Caverlee, J.; and Lee, K. 2010. You Are Where You Tweet: A Content-Based Approach to Geo-locating Twitter Users. In Proceedings of The 19th ACM Inter- national Conference on Information and Knowledge Man- agement, 759-768. Toronto, Ontario, Canada: International Conference on Information and Knowledge Management.
- Costenbader, E., and Valente, T. W. 2003. The stability of centrality measures when networks are sampled. Social networks 25(4):283-307.
- Cover, T. M., and Thomas, J. A. 2006. Elements of Infor- mation Theory. Hoboken, New Jersey: Wiley InterScience.
- De Choudhury, M.; Lin, Y.-R.; Sundaram, H.; Candan, K. S.; Xie, L.; and Kelliher, A. 2010. How Does the Data Sampling Strategy Impact the Discovery of Information Diffusion in Social Media. In Proc. of the 4th Int'l AAAI Conference on Weblogs and Social Media, 34-41. Washington, DC, USA: AAAI.
- De Longueville, B.; Smith, R. S.; and Luraschi, G. 2009. "omg, from here, i can see the flames!": a use case of mining location based social networks to acquire spatio-temporal data on forest fires. In Proceedings of the 2009 Interna- tional Workshop on Location Based Social Networks, LBSN '09, 73-80. New York, NY, USA: ACM.
- Dumais, S. T.; Furnas, G. W.; Landauer, T. K.; Deerwester, S.; and Harshman, R. 1988. Using latent semantic analysis to improve access to textual information. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '88, 281-285. New York, NY, USA: ACM.
- Efron, M. 2010. Hashtag retrieval in a microblogging en- vironment. In Proc. of the 33rd international ACM SIGIR conference on Research and development in information re- trieval, SIGIR '10, 787-788. New York, NY, USA: ACM.
- Freeman, L. C. 1979. Centrality in Social Networks: Con- ceptual clarification. Social Networks 1(3):215-239.
- Galil, Z. 1986. Efficient algorithms for finding maximum matching in graphs. ACM Comput. Surv. 18(1):23-38.
- Gayo-Avello, D.; Metaxas, P. T.; and Mustafaraj, E. 2011. Limits of electoral predictions using twitter. In Proceedings of the International Conference on Weblogs and Social Me- dia, volume 21, 490-493. Barcelona, Spain: AAAI.
- Granovetter, M. 1976. Network Sampling: Some First Steps. American Journal of Sociology 81(6):1287-1303.
- Hong, L., and Davison, B. D. 2010. Empirical study of topic modeling in twitter. In Proc. First Workshop on Social Me- dia Analytics, SOMA '10, 80-88. NYC, NY, USA: ACM.
- Hong, L.; Ahmed, A.; Gurumurthy, S.; Smola, A. J.; and Tsioutsiouliklis, K. 2012. Discovering geographical topics in the twitter stream. In Proceedings of the 21st interna- tional conference on World Wide Web, WWW '12, 769-778.
- Joseph, K.; Tan, C. H.; and Carley, K. M. 2012. Beyond "lo- cal", "categories" and "friends": clustering foursquare users with latent "topics". In Proceedings of the 2012 ACM Con- ference on Ubiquitous Computing, UbiComp '12, 919-926.
- Kireyev, K.; Palen, L.; and Anderson, K. 2009. Applications of Topics Models to Analysis of Disaster-Related Twitter Data. In NIPS Workshop on Applications for Topic Models: Text and Beyond, volume 1.
- Kossinets, G. 2006. Effects of missing data in social net- works. Social Networks 28(3):247-268.
- Kumar, S.; Barbier, G.; Abbasi, M. A.; and Liu, H. 2011. Tweettracker: An analysis tool for humanitarian and disaster relief. In Fifth International AAAI Conference on Weblogs and Social Media. Barcelona Spain: AAAI.
- Leskovec, J., and Faloutsos, C. 2006. Sampling from Large Graphs. In Proceedings of the 12th ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Min- ing, 631-636.
- Lin, J. Jan. Divergence measures based on the shan- non entropy. Information Theory, IEEE Transactions on 37(1):145-151.
- Mathioudakis, M., and Koudas, N. 2010. Twittermonitor: trend detection over the twitter stream. In Proc. of the 2010 ACM SIGMOD Int'l Conference on Management of data, SIGMOD '10, 1155-1158. New York, NY, USA: ACM.
- Pozdnoukhov, A., and Kaiser, C. 2011. Space-time dynam- ics of topics in streaming text. In Proc. of the 3rd ACM SIGSPATIAL Int'l Workshop on Location-Based Social Net- works, LBSN '11, 1-8. New York, NY, USA: ACM.
- Recuero, R., and Araujo, R. 2012. On the rise of artificial trending topics in twitter. In Proceedings of the 23rd ACM conference on Hypertext and social media, HT '12, 305- 306. New York, NY, USA: ACM.
- Řehůřek, R., and Sojka, P. 2010. Software Framework for Topic Modelling with Large Corpora. In Proc. of the LREC 2010 Workshop on New Challenges for NLP Frameworks, 45-50. Valletta, Malta: ELRA.
- Sofean, M., and Smith, M. 2012. A real-time architecture for detection of diseases using social networks: design, imple- mentation and evaluation. In Proceedings of the 23rd ACM conference on Hypertext and social media, HT '12, 309- 310. New York, NY, USA: ACM.
- Tsur, O., and Rappoport, A. 2012. What's in a hashtag?: content based prediction of the spread of ideas in microblog- ging communities. In Proceedings of the fifth ACM interna- tional conference on Web search and data mining, WSDM '12, 643-652. New York, NY, USA: ACM.
- Tumasjan, A.; Sprenger, T. O.; Sandner, P. G.; and Welpe, I. M. 2010. Predicting elections with twitter: What 140 characters reveal about political sentiment. In Proceedings of the International Conference on Weblogs and Social Me- dia, 178-185. Washington, DC, USA: AAAI.
- Wakamiya, S.; Lee, R.; and Sumiya, K. 2011. Crowd-based urban characterization: extracting crowd behavioral patterns in urban areas from twitter. In Proc. of the 3rd ACM SIGSPA- TIAL Int'l Workshop on Location-Based Social Networks, LBSN '11, 77-84. New York, NY, USA: ACM.
- Wasserman, S., and Faust, K. 1994. Social Network Analy- sis: Methods and Applications. Cambridge, MA: Cambridge University Press.
- Yang, L.; Sun, T.; Zhang, M.; and Mei, Q. 2012. We know what @you #tag: does the dual role affect hashtag adoption? In Proc. of the 21st int'l conference on World Wide Web, WWW '12, 261-270. New York, NY, USA: ACM.
- Yin, Z.; Cao, L.; Han, J.; Zhai, C.; and Huang, T. 2011. Geographical topic discovery and comparison. In Proceed- ings of the 20th international conference on World wide web, WWW '11, 247-256. New York, NY, USA: ACM.
Cited by
Sorted by citations
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
Self-citationOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 928 · see citing works
“For each of these accounts, we collected their tweets through the Twitter Search API.”
MethodInfluentialSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Disinformation and social bot operations in the run up to the 2017 French presidential election
Ferrara, E. (2017) · First Monday · doi:10.5210/fm.v22i8.8005 · Cited by 524 · see citing works
Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media
ElSherief, M., Kulkarni, V., Nguyen, D., Yang Wang, W., & Belding, E. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15041 · Cited by 294 · see citing works
Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign
Keller, F. B., Schoch, D., Stier, S., & Yang, J. (2020) · Political Communication · doi:10.1080/10584609.2019.1661888 · Cited by 378 · see citing works
Assessing the bias in samples of large online networks
González-Bailón, S., Wang, N., Rivero, A., Borge-Holthoefer, J., & Moreno, Y. (2014) · Social Networks · doi:10.1016/j.socnet.2014.01.004 · Cited by 238 · see citing works
What types of COVID-19 conspiracies are populated by Twitter bots?
Ferrara, E. (2020) · First Monday · doi:10.5210/fm.v25i6.10633 · Cited by 223 · see citing works
“nly observing a small fraction, approximately 1%, of the overall Twitter conversation, through the lens of the Twitter API. This has been shown to introduce some biases toward over-represented topics [87], and COVID-19 has been the most spoken-about topic of discussion ever since the beginning of the pandemic. Second,…”
BackgroundTampering with Twitter’s Sample API
Pfeffer, J., Mayer, K., & Morstatter, F. (2018) · EPJ Data Science · doi:10.1140/epjds/s13688-018-0178-0 · Cited by 135 · see citing works
Self-citationFischer-Preßler, D., Schwemmer, C., & Fischbach, K. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.012 · Cited by 133 · see citing works
Who Falls for Online Political Manipulation?
Badawy, A., Lerman, K., & Ferrara, E. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3316494 · Cited by 105 · see citing works
When is it biased?: assessing the representativeness of twitter's streaming API
Morstatter, F., Pfeffer, J., & Liu, H. (2014) · Proceedings of the 23rd International Conference on World Wide Web · doi:10.1145/2567948.2576952 · Cited by 90 · see citing works
Self-citationSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
Investigating Italian disinformation spreading on Twitter in the context of 2019 European elections
Pierri, F., Artoni, A., & Ceri, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227821 · Cited by 93 · see citing works
Predicting Elections for Multiple Countries Using Twitter and Polls
Tsakalidis, A., Papadopoulos, S., Cristea, A. I., & Kompatsiaris, Y. (2015) · IEEE Intelligent Systems · doi:10.1109/mis.2015.17 · Cited by 79 · see citing works
Two 1%s don't make a whole: comparing simultaneous samples from twitter's streaming api
Joseph, K., Landwehr, P. M., & Carley, K. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-05579-4_10 · Cited by 69 · see citing works
Self-citationThe Effect of Collective Attention on Controversial Debates on Social Media
Garimella, K., De Francisci Morales, G., Gionis, A., & Mathioudakis, M. (2017) · Proceedings of the 2017 ACM on Web Science Conference · doi:10.1145/3091478.3091486 · Cited by 56 · see citing works
“6 Admittedly, this dataset might suffer from sampling bias, however the topics are specific enough that the distortion should be negligible [28].”
BackgroundLiang, H., & Fu, K.-W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0134270 · Cited by 42 · see citing works
Ramadona, A. L., Tozan, Y., Lazuardi, L., & Rocklöv, J. (2019) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0007298 · Cited by 42 · see citing works
Affective Polarization in Online Climate Change Discourse on Twitter
Tyagi, A., Uyheng, J., & Carley, K. M. (2020) · 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam49781.2020.9381419 · Cited by 41 · see citing works
Self-citationCross-Platform Reactions to the Post-January 6 Deplatforming
Buntain, C., Innes, M., Mitts, T., & Shapiro, J. (2023) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2023.004 · Cited by 25 · see citing works
Kingdom of Trolls? Influence Operations in the Saudi Twittersphere
Barrie, C., & Siegel, A. (2021) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2021.012 · Cited by 12 · see citing works
“Recent research on the Twitter sampling API reveals that it is vulnerable to artificial distortion and may not represent an actual 1% random sample (Morstatter et al., 2013; Pfeffer et al., 2018).”
Background
- Authors
- Morstatter, F., Pfeffer, J., Liu, H., & Carley, K.
- Affiliations
- Arizona State University (US)
- Carnegie Mellon University (US)
- Venue
- Proceedings of the International AAAI Conference on Web and Social Media
- Published
- 2021-08-03
- Volume
- 7
- Issue
- 1
- Pages
- 400-408
- ISSN
- 2162-3449 (ISSN-L)
- 2334-0770
- OpenAlex
- W1845748792
- Cited by
- 863
- References
- 0
- Citations by year
- 2026: 18 · 2025: 38 · 2024: 15 · 2023: 34 · 2022: 39 · 2021: 62 · 2020: 88 · 2019: 80 · 2018: 90 · 2017: 92 · 2016: 109 · 2015: 90 · 2014: 86 · 2013: 20 · 2012: 1
- FWCI
- 96.37
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Office of Naval Research (N000141110527, N000141010091, N000141010091 and N000141110527)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Complex Network Analysis Techniques
- Social Sciences › Communication › Social Media and Politics
- Keywords
- Twitter, sampling bias, microblogging, social media data, data comparison, API documentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
