The DOI matches its Crossref record; 78 of the 82 references it cites were found in a registry or an index.
Tampering with Twitter’s Sample API
Open accessAbstract
Social media data is widely analyzed in computational social science. Twitter, one of the largest social media platforms, is used for research, journalism, business, and government to analyze human behavior at scale. Twitter offers data via three different Application Programming Interfaces (APIs). One of which, Twitter’s Sample API, provides a freely available 1% and a costly 10% sample of all Tweets. These data are supposedly random samples of all platform activity. However, we demonstrate that, due to the nature of Twitter’s sampling mechanism, it is possible to deliberately influence these samples, the extent and content of any topic, and consequently to manipulate the analyses of researchers, journalists, as well as market and political analysts trusting these data sources. Our analysis also reveals that technical artifacts can accidentally skew Twitter’s samples. Samples should therefore not be regarded as random. Our findings illustrate the critical limitations and general issues of big data sampling, especially in the context of proprietary data and undisclosed details about data handling.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Such accounts use tricks such as hijacking “trending” hashtags of keywords to gain visibility (Thomas et al., 2011), and can (purposefully or accidentaly) distort the statistics of datasets collected from social platforms”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Pfeffer et al., 2018
In the citing paper
“However, the Twitter Streaming API is like a black-box with a lack of transparency in the sampling methodology [41,42], yet this is one of the few forms of unfettered and large-scale social media data access to researchers outside social media companies [43] and has been used in prior research, including in health-related studies [35,44,45].”
From Psychosocial Effects of the COVID-19 Pandemic: Large-scale Quasi-Experimental Study on Social Media (2020) · cites this paper as [41]
In the citing paper
“Recent research has also questioned the credibility of the “1% Twitter stream” aspect, noting that actual sampling data was smaller than what it ideally should have been [41].”
From Psychosocial Effects of the COVID-19 Pandemic: Large-scale Quasi-Experimental Study on Social Media (2020) · cites this paper as [41]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957
Opinion mining and sentiment analysis
Pang, B., & Lee, L. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000011 · Cited by 6,372
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Big data: a revolution that will transform how we live, work, and think
(2013) · Choice Reviews Online · doi:10.5860/choice.50-6804 · Cited by 3,657
Earthquake shakes Twitter users
Sakaki, T., Okazaki, M., & Matsuo, Y. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772777 · Cited by 3,631
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Twitter as a Corpus for Sentiment Analysis and Opinion Mining
Pak, A., & Paroubek, P. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/47akcj8foo4d · Cited by 2,784
Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment
Tumasjan, A., Sprenger, T. O., Sandner, P., & Welpe, I. M. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14009 · Cited by 2,647
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523
Tweet, Tweet, Retweet: Conversational Aspects of Retweeting on Twitter
Boyd, D., Golder, S., & Lotan, G. (2010) · 2010 43rd Hawaii International Conference on System Sciences · doi:10.1109/hicss.2010.412 · Cited by 2,051
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series
O'Connor, B., Balasubramanyan, R., Routledge, B., & Smith, N. A. (2010) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v4i1.14031 · Cited by 1,949
Big data: a revolution that will transform how we live, work, and think
Armstrong, K. (2014) · Information Communication & Society · doi:10.1080/1369118x.2014.923482 · Cited by 1,645
Sentiment Analysis of Twitter Data
Agarwal, A., Xie, B., Vovsha, I., Rambow, O., & Passonneau, R. J. (2011) · Cited by 1,565
Finding high-quality content in social media
Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1341531.1341557 · Cited by 1,255
The political power of social media: Technology, the public sphere, and political change
Shirky, C. (2011) · Foreign Affairs · Cited by 1,193
Twitter adoption and use in mass convergence and emergency events
Hughes, A., & Palen, L. (2009) · International Journal of Emergency Management · doi:10.1504/ijem.2009.031564 · Cited by 1,002
Online Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981
SemEval-2017 Task 4: Sentiment Analysis in Twitter
Rosenthal, S., Farra, N., & Nakov, P. (2017) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s17-2088 · Cited by 930
Is the Sample Good Enough? Comparing Data from Twitter's Streaming API with Twitter's Firehose
Morstatter, F., Pfeffer, J., Liu, H., & Carley, K. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v7i1.14401 · Cited by 863
Self-citationLasorsa, D. L., Lewis, S. C., & Holton, A. E. (2011) · Journalism Studies · doi:10.1080/1461670x.2011.571825 · Cited by 833
"I always assumed that I wasn't really that close to [her]"
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K. G., Hamilton, K., & Sandvig, C. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 781
Tufekci, Z. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14517 · Cited by 780
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Understanding the Demographics of Twitter Users
Mislove, A. E., Lehmann, S., Ahn, Y., Onnela, J.-P. “., & Rosenquist, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14168 · Cited by 761
Social media for large studies of behavior
Ruths, D., & Pfeffer, J. (2014) · Science · doi:10.1126/science.346.6213.1063 · Cited by 648
Self-citationDetecting and Tracking Political Abuse in Social Media
Ratkiewicz, J., Conover, M., Meiss, M., Gonçalves, B., Flammini, A., & Menczer, F. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14127 · Cited by 576
Proceedings of the 26th International Conference on World Wide Web - WWW '17
(2017) · doi:10.1145/3038912 · Cited by 555
A System for Real-time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle
Wang, H., Can, D., Kazemzadeh, A., Bar, F., & Narayanan, S. S. (2012) · Cited by 542
Chu, Z., Gianvecchio, S., Wang, H., & Jajodia, S. (2010) · Annual Computer Security Applications Conference · doi:10.1145/1920261.1920265 · Cited by 498
Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter
Lee, K., Eoff, B. D., & Caverlee, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14106 · Cited by 461
THE REUTERS INSTITUTE FOR THE STUDY OF JOURNALISM
Levy, D. A. L. (2011) · Journalism Studies · doi:10.1080/1461670x.2010.530965 · Cited by 454
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405
Ensuring the data-rich future of the social sciences
King, G. (2011) · Science · doi:10.1126/science.1197872 · Cited by 394
Proceedings of the 12th international conference on World Wide Web
Hencsey, G., & Association for Computing Machinery-Digital Library (2003) · Cited by 373
Validity Issues in the Use of Social Network Analysis with Digital Trace Data
University of Texas at Austin, Howison, J., Wiggins, A., Syracuse University, Crowston, K., & Syracuse University (2011) · Journal of the Association for Information Systems · doi:10.17705/1jais.00282 · Cited by 369
A Meta-Analysis of State-of-the-Art Electoral Prediction From Twitter Data
Gayo-Avello, D. (2013) · Social Science Computer Review · doi:10.1177/0894439313493979 · Cited by 315
Crisis informatics—New data for extraordinary times
Palen, L., & Anderson, K. M. (2016) · Science · doi:10.1126/science.aag2579 · Cited by 309
Peeking Beneath the Hood of Uber
Chen, L., Mislove, A. E., & Wilson, C. (2015) · ACM Internet Measurement Conference (IMC) · doi:10.1145/2815675.2815681 · Cited by 293
Anatomy of an online misinformation network
Shao, C., Hui, P.-M., Wang, L., Jiang, X., Flammini, A., Menczer, F., & Ciampaglia, G. L. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196087 · Cited by 282
A topology of Twitter research: disciplines, methods, and ethics
Zimmer, M., & Proferes, N. J. (2014) · Aslib Journal of Information Management · doi:10.1108/ajim-09-2013-0083 · Cited by 257
Ausserhofer, J., & Maireder, A. (2013) · Information Communication & Society · doi:10.1080/1369118x.2012.756050 · Cited by 252
Proceedings of the 2008 International Conference on Web Search and Data Mining
Najork, M., Broder, A., & Chakrabarti, S. (2008) · Cited by 250
Quantitative Approaches to Comparing Communication Patterns on Twitter
Bruns, A., & Stieglitz, S. (2012) · Journal of Technology in Human Services · doi:10.1080/15228835.2012.744249 · Cited by 249
The Relative Importance of Social Media for Accessing, Finding, and Engaging with News
Nielsen, R., & Schrøder, K. (2014) · Digital Journalism · doi:10.1080/21670811.2013.872420 · Cited by 240
Proceedings of ACL 2018, System Demonstrations
(2018) · doi:10.18653/v1/p18-4 · Cited by 238
Assessing the bias in samples of large online networks
González-Bailón, S., Wang, N., Rivero, A., Borge-Holthoefer, J., & Moreno, Y. (2014) · Social Networks · doi:10.1016/j.socnet.2014.01.004 · Cited by 238
Critiquing Big Data: Politics, Ethics, Epistemology
Crawford, K., Miltner, K. M., & Gray, M. L. (2014) · International journal of communication · Cited by 235
What do people study when they study Twitter? Classifying Twitter related academic papers
Williams, S. A., Terras, M., & Warwick, C. (2013) · Journal of Documentation · doi:10.1108/jd-03-2012-0027 · Cited by 223
Online social networks and offline protest
STEINERT-THRELKELD, Z., Mocanu, D., Vespignani, A., & Fowler, J. H. (2015) · EPJ Data Science · doi:10.1140/epjds/s13688-015-0056-y · Cited by 208
(2018) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508306 · Cited by 197
Crawford, K., & Finn, M. (2015) · GeoJournal · doi:10.1007/s10708-014-9597-z · Cited by 191
Wang, W., Chen, L., Thirunarayan, K., & Sheth, A. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531734 · Cited by 175
Population Bias in Geotagged Tweets
Malik, M. M., Lamba, H., Nakos, C., & Pfeffer, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i4.14688 · Cited by 172
Self-citationProceedings of the Workshop on Languages in Social Media
Nagarajan, M., & Gamon, M. (2011) · Cited by 142
Early detection of promoted campaigns on social media
Varol, O., Ferrara, E., Menczer, F., & Flammini, A. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0111-y · Cited by 133
Compromised data: from social media to big data
(2016) · Choice Reviews Online · doi:10.5860/choice.194921 · Cited by 126
Mining One Percent of Twitter: Collections, Baselines, Sampling
Gerlitz, C., & Rieder, B. (2013) · M/C Journal · doi:10.5204/mcj.620 · Cited by 114
Bastos, M. T. (2014) · Journalism Studies · doi:10.1080/1461670x.2014.891857 · Cited by 97
Are Social Bots on Twitter Political Actors? Empirical Evidence from a Ukrainian Social Botnet
Hegelich, S., & Janetzko, D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14764 · Cited by 96
When is it biased?: assessing the representativeness of twitter's streaming API
Morstatter, F., Pfeffer, J., & Liu, H. (2014) · Proceedings of the 23rd International Conference on World Wide Web · doi:10.1145/2567948.2576952 · Cited by 90
Self-citationDriscoll, K., & Walker, S. (2014) · International journal of communication · Cited by 79
Early filtering of ephemeral malicious accounts on Twitter
Lee, S., & Kim, J. K. (2014) · Computer Communications · doi:10.1016/j.comcom.2014.08.006 · Cited by 76
Two 1%s don't make a whole: comparing simultaneous samples from twitter's streaming api
Joseph, K., Landwehr, P. M., & Carley, K. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-05579-4_10 · Cited by 69
Easy data, hard data: The politics and pragmatics of Twitter research after the computational turn
Burgess, J., & Bruns, A. (2015) · Bloomsbury Academic eBooks · doi:10.5040/9781501306549.0010 · Cited by 65
Entropy-based outlier detection using semi-supervised approach with few positive examples
Daneshpazhouh, A., & Sami, A. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.06.012 · Cited by 61
Discovery, Retrieval, and Analysis of the 'Star Wars' Botnet in Twitter
Echeverría, J., & Zhou, S. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3110074 · Cited by 58
Compromised Data : From Social Media to Big Data
(2015) · Bloomsbury Academic eBooks · doi:10.5040/9781501306549 · Cited by 51
A Biased Review of Biases in Twitter Studies on Political Collective Action
Cihon, P., & Yasseri, T. (2016) · Frontiers in Physics · doi:10.3389/fphy.2016.00034 · Cited by 46
Identifying platform effects in social media data
Malik, M., & Pfeffer, J. (2016) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14756 · Cited by 45
Self-citationSampling from Social Networks with Attributes
Wagner, C., Singer, P., Karimi, F., Pfeffer, J., & Strohmaier, M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3038912.3052665 · Cited by 43
Self-citationOn the endogenesis of Twitter's Spritzer and Gardenhose sample streams
Kergl, D., Roedler, R., & Seeber, S. (2014) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2014.6921610 · Cited by 28
Can One Tamper with the Sample API?: Toward Neutralizing Bias from Spam and Bot Content
Morstatter, F., Dani, H., Sampson, J., & Liu, H. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889372 · Cited by 24
Self-citationzooRank: Ranking Suspicious Entities in Time-Evolving Tensors
Lamba, H., Hooi, B., Shin, K., Faloutsos, C., & Pfeffer, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71249-9_5 · Cited by 6
Self-citationCooking Up Healthy Citizens: The Pedagogy of Cookbooks
Gray, E., & Leahy, D. (2013) · M/C Journal · doi:10.5204/mcj.645 · Cited by 3
Effects of Sampling on Twitter Trend Detection
Yates, A., Kolcz, A., Goharian, N., & Frieder, O. (2016) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/496yiwq6afa6 · Cited by 2
Research works printed but not linked: 6
- Elmer G, Langlois G, Redden J (2015) Compromised data: from social media to big data. Bloomsbury Publishing, New York
- Gaffney D, Puschmann C (2013) Data collection on Twitter, pp. 55–67. Peter Lang, New York
- Neuberger C, vom Hofe J, Nuernbergk C (2014) The use of Twitter by professional journalists. results of a newsroom survey in Germany. In: Weller K, Bruns A, Burgess J, Mahrt M, Puschmann C (eds) Twitter and society. Peter Lang, New York, pp 345–357
- Elmer G, Langlois G, Redden J (2015) Compromised data: from social media to big data. Bloomsbury Publishing, USA
- Weller K, Bruns A, Burgess J, Mahrt M, Puschmann C Twitter and Society, vol. 89. P. Lang
- de Saint-Exupéry A (1943) The Little Prince. Reynal & Hitchcock, New York
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
“, 2011), and can (purposefully or accidentaly) distort the statistics of datasets collected from social platforms (Morstatter et al., 2016; Pfeffer et al., 2018).”
BackgroundAssessing the risks of ‘infodemics’ in response to COVID-19 epidemics
Gallotti, R., Valle, F., Castaldo, N., Sacco, P., & De Domenico, M. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-00994-6 · Cited by 591
Sentiment analysis of Twitter data during critical events through Bayesian networks classifiers
Ruz, G. A., Henríquez, P. A., & Mascareño, A. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.01.005 · Cited by 348
Psychosocial Effects of the COVID-19 Pandemic: Large-scale Quasi-Experimental Study on Social Media
Saha, K., Torous, J., Caine, E. D., & De Choudhury, M. (2020) · Journal of Medical Internet Research · doi:10.2196/22600 · Cited by 166
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Metadata
- Authors
- Pfeffer, J., Mayer, K., & Morstatter, F.
- Affiliations
- Technical University of Munich (DE)
- University of Southern California (US)
- Corresponding author
- Pfeffer, Jürgen
- Venue
- EPJ Data Science
- Published
- 2018-12
- Volume
- 7
- Issue
- 1
- ISSN
- 2193-1127 (ISSN-L)
- OpenAlex
- W2904605268
- Cited by
- 135
- References
- 82
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 7 · 2023: 22 · 2022: 22 · 2021: 25 · 2020: 31 · 2019: 10
- FWCI
- 23.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- EPJ Data Science (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- Funders
- Deutsche Forschungsgemeinschaft
- Technische Universität München
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- sampling bias, tweets, data manipulation, random sampling, technical artifacts, proprietary data, social media data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
