When is it biased?: assessing the representativeness of twitter's streaming API
Abstract
Twitter shares a free 1% sample of its tweets through the "Streaming API". Recently, research has pointed to evidence of bias in this source. The methodologies proposed in previous work rely on the restrictive and expensive Firehose to find the bias in the Streaming API data. We tackle the problem of finding sample bias without costly and restrictive Firehose data. We propose a solution that focuses on using an open data source to find bias in the Streaming API.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the case of Twitter, much research relies on APIs that give access to at most 1% of the data.
“Further, in the case of Twitter, much research relies on APIs that give access to at most 1% of the data”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Morstatter et al., 2014
The Twitter Streaming API lacks transparency in its sampling methodology.
“However, the Twitter Streaming API is like a black-box with a lack of transparency in the sampling methodology [41,42], yet this is one of the few forms of unfettered and large-scale social media data access to researchers outside social media companies [43] and has been used in prior research, including in health-related studies [35,44,45].”
From Psychosocial Effects of the COVID-19 Pandemic: Large-scale Quasi-Experimental Study on Social Media (2020) · cites this paper as [42]
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
The Jackknife, the Bootstrap and Other Resampling Plans
Efron, B. (1982) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611970319 · Cited by 8,120 · see citing works
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
Empirical study of topic modeling in Twitter
Hong, L., & Davison, B. D. (2010) · doi:10.1145/1964858.1964870 · Cited by 1,142 · see citing works
Cheng, Z., Caverlee, J., & Lee, K. (2010) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1871437.1871535 · Cited by 1,085 · see citing works
Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose
Morstatter, F., Pfeffer, J., Liu, H., & Carley, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v7i1.14401 · Cited by 863 · see citing works
Self-citationMathioudakis, M., & Koudas, N. (2010) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/1807167.1807306 · Cited by 824 · see citing works
Understanding the Demographics of Twitter Users
Mislove, A. E., Lehmann, S., Ahn, Y., Onnela, J.-P. “., & Rosenquist, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14168 · Cited by 761 · see citing works
Statistical Methods for Quality Improvement
SRJ, & Ryan, T. P. (2000) · Journal of the American Statistical Association · doi:10.2307/2669810 · Cited by 755 · see citing works
Connecting users across social media sites
Zafarani, R., & Liu, H. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2487648 · Cited by 408 · see citing works
Self-citationHow does the data sampling strategy impact the discovery of information diffusion in social media?
De Choudhury, M., Lin, Y.-R., Sundaram, H., Candan, K. S., Xie, L., & Kelliher, A. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14024 · Cited by 209 · see citing works
Characterizing the Life Cycle of Online News Stories Using Social Media Reactions
Castillo, C., El-Haddad, M., Jazeera, A., Stempeck, M., & Pfeffer, J. (2013) · Cited by 181 · see citing works
Self-citationSpace-time dynamics of topics in streaming text
Pozdnoukhov, A., & Kaiser, C. (2011) · Workshop on Location-based Social Networks · doi:10.1145/2063212.2063223 · Cited by 90 · see citing works
Gaussian process regression bootstrapping: exploring the effects of uncertainty in time course data
Kirk, P., & Stumpf, M. P. H. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp139 · Cited by 83 · see citing works
A real-time architecture for detection of diseases using social networks
Sofean, M., & Smith, M. S. (2012) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/2309996.2310048 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“Further, in the case of Twitter, much research relies on APIs that give access to at most 1% of the data (González-Bailón et al., 2014a; Joseph et al., 2014; Morstatter et al., 2014).”
BackgroundSocial media for large studies of behavior
Ruths, D., & Pfeffer, J. (2014) · Science · doi:10.1126/science.346.6213.1063 · Cited by 648 · see citing works
Self-citationPsychosocial Effects of the COVID-19 Pandemic: Large-scale Quasi-Experimental Study on Social Media
Saha, K., Torous, J., Caine, E. D., & De Choudhury, M. (2020) · Journal of Medical Internet Research · doi:10.2196/22600 · Cited by 166 · see citing works
Tampering with Twitter’s Sample API
Pfeffer, J., Mayer, K., & Morstatter, F. (2018) · EPJ Data Science · doi:10.1140/epjds/s13688-018-0178-0 · Cited by 135 · see citing works
Self-citationSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
- Authors
- Morstatter, F., Pfeffer, J., & Liu, H.
- Affiliations
- Arizona State University (US)
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 23rd International Conference on World Wide Web
- Publisher
- ACM
- Published
- 2014-04-07
- Pages
- 555-556
- OpenAlex
- W1917841893
- Cited by
- 90
- References
- 16
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 4 · 2022: 6 · 2021: 11 · 2020: 12 · 2019: 12 · 2018: 10 · 2017: 10 · 2016: 12 · 2015: 10 · 2014: 1
- FWCI
- 76.50
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International World Wide Web Conference (WWW) (publishedVersion)
- Funders
- Office of Naval Research (N000141010091 and N000141110527)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- Twitter Streaming API, sampling bias, open data sources, social media data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
