CoChat
Sign Up
Cited by 131Social Media and PoliticsSentiment Analysis and Opinion MiningSurvey Methodology and Nonresponse

Online and Social Media Data As an Imperfect Continuous Panel Survey

Open access
Diaz, Fernando, Gamon, Michael, Hofman, Jake M., Kıcıman, Emre, Rothschild, David·2016·PLOS ONE
DOI 10.1371/journal.pone.0145406PMC PMC4711590
Open this paper in CoChat — free accountOpen PDF

Abstract

There is a large body of research on utilizing online activity as a survey of political opinion to predict real world election outcomes. There is considerably less work, however, on using this data to understand topic-specific interest and opinion amongst the general population and specific demographic subgroups, as currently measured by relatively expensive surveys. Here we investigate this possibility by studying a full census of all Twitter activity during the 2012 election cycle along with the comprehensive search history of a large panel of Internet users during the same period, highlighting the challenges in interpreting online and social media activity as the results of a survey. As noted in existing work, the online population is a non-representative sample of the offline world (e.g., the U.S. voting population). We extend this work to show how demographic skew and user participation is non-stationary and difficult to predict over time. In addition, the nature of user contributions varies substantially around important events. Furthermore, we note subtle problems in mapping what people are sharing or consuming online to specific sentiment or opinion measures around a particular topic. We provide a framework, built around considering this data as an imperfect continuous panel survey, for addressing these issues so that meaningful insight about public interest and opinion can be reliably extracted from online and social media data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0145406

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • By selecting political tweets during the 2012 US election, Diaz et al. noticed a user population biased toward Washington, DC.

    “by selecting political tweets during the 2012 US election, Diaz et al. (2016) noticed a user population biased toward Washington, DC”

    From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Diaz et al., 2016

    The Data section reports that Mid-Atlantic election-related Twitter activity was overrepresented relative to voters, and the region includes Washington, DC. But it measures engagement across the region, not a DC-specific bias in the user population. Page numbers are unavailable in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • User demographic composition and participation on a specific topic or interaction patterns are often non-stationary.

    “Even the user demographic composition and participation on a specific topic (Guerra et al., 2014; Diaz et al., 2016) or their interaction patterns (Viswanath et al., 2009) are often non-stationary.”

    From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Diaz et al., 2016

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 74 · Cited by 131

Research works printed but not linked: 51

  1. [2]O'Connor B, Balasubramanyan R, Routledge B, Smith NA. From tweets to polls: Linking text sentiment to public opinion time series. ICWSM. 2010; 11: 122– 129.
  2. [3]DiGrazia J, McKelvey K, Bollen J, Rojas F. More tweets, more votes: Social media as a quantitative indicator of political behavior. PLoS ONE. 2013; 8 11: e79449 10.1371/journal.pone.0079449 24312181 PMC3842288 · DOI
  3. [5]Gayo-Avello D. Don't turn social media into another Literary Digest poll. Communications of the ACM. 2011; 54 10: 121– 128.
  4. [7]Lui C, Metaxas PT, Mustafaraj E. On the predictability of the US elections through search volume activity. Proceedings of the IADIS International Conference on e-Society. 2011.
  5. [8]Biemer P. Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly. 2010; 74(5): 817– 848.
  6. [9]Groves RM, Lyberg L. Total survey error: Past, present, and future. Public Opinion Quarterly. 2010; 74(5): 849– 879.
  7. [10]Pew Research Assessing the Representativeness of Public Opinion Surveys. Pew Research Center. 2012. Available: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-public-opinion-surveys/.
  8. [12]Budak C, Watts DJ. Dissecting the Spirt of Gezi: Influence vs. selection in the Occupy Gezi movement. Sociological Science. 2015; 10.15195/v2.a18 · DOI
  9. [14]Ratkiewicz J, Conover M, Meiss M, Bruno G, Patil S, Flammini A, et al. Truthy: mapping the spread of astroturf in microblog streams. Proceedings of the 20th international conference companion on World wide web 2011.
  10. [15]Ginsberg J, Mohebbi MH. Detecting influenza epidemics using search engine query data. Nature. 2009; 457 7232: 1012– 1014. 10.1038/nature07634 19020500 · DOI · Paper page
  11. [18]Dodds PS, Harris KD, Kloumann I, Bliss CA, Danforth CM. Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. PLoS ONE. 2011; 6 12: e26752: 10.1371/journal.pone.0026752 22163266 PMC3233600 · DOI · Paper page
  12. [21]Moe H, Larsson A. Methodological and ethical challenges associated with large–scale analyses of online political communication. Nordicom Review. 2012.
  13. [23]Vallina-Rodriguez N, Scellato S, Haddadi H, Carl F, Crowcroft J, Mascolo C. Los twindignados: The rise of the indignados movement on twitter. Privacy, Security, Risk and Trust PASSAT, 2012 International Conference on and 2012 International Confernece on Social Computing SocialCom 2012.
  14. [26]Schober MF, Pasek J, Guggenheim L, Cliff L, Conrad FB. Research synthesis: Social media analyses for social measurement. Public Opinion Quarterly (under revision). 2015. Available: http://joshpasek.com/. PMC4884815 27257310
  15. [27]Conover MD, Ferrara E, Menczer F, Flammini A. The digital evolution of occupy wall street. PLoS ONE. 2013; 8 5: e64679 23734215 PMC3667169
  16. [28]Huberman BA, Romero DM, Wu F. Social networks that matter: Twitter under the microscope. arXiv preprint arXiv:0812.1045. 2008. · arXiv
  17. [29]Soler JM, Cuartero F, Roblizo M. Twitter as a Tool for Predicting Elections Results. Proceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining ASONAM 2012 2012.
  18. [30]Skoric M, Poor N, Achananuparp P, Ee-Peng L, Jiang J. Tweets and votes: A study of the 2011 Singapore general election. System Science HICSS, 2012 45th Hawaii International Conference 2012.
  19. [31]Bermingham A, Smeaton AF. On using Twitter to monitor political sentiment and predict election results. Sentiment Analysis where AI meets Psychology (SAAIP) Workshop at the International Joint Conference for Natural Language Processing (IJCNLP). 2011.
  20. [33]Monti C, Zignani M, Rozza A, Arvidsson A, Zappella G, Colleoni E. Modelling political disaffection from Twitter data. Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining. 2013.
  21. [34]Choy M, Cheong M, Laik MN, Shung KP. US Presidential Election 2012 Prediction using Census Corrected Twitter Model. arXiv preprint arXiv:1211.0938. 2012. · arXiv
  22. [35]Shi L, Agarwal N, Agrawal A, Rahul G, Spoelstr J. Predicting US primary elections with Twitter. Proceedings of Social Network and Social Media Analysis: Methods, Models and Applications NIPS Workshop, Lake Tahoe, NV, December Vol. 7. 2012.
  23. [37]He Y, Saif H, Wei Z, Wong KF. Quantising Opinions for Political Tweets Analysis. LREC. 2012; 3901–3906.
  24. [39]Jungherr A, Jürgens P, Schoen H. Why the pirate party won the german election of 2009 or the trouble with predictions: A response to tumasjan, a., sprenger, to, sander, pg, & welpe, im “predicting elections with twitter: What 140 characters reveal about political sentiment. Social Science Computer Review 2012.
  25. [40]Metaxas PT, Mustafaraj E. Social media and the elections. Science. 2012; 3386106: 472– 473. 23112315
  26. [43]Mislove A, Lehmann S, Ahn Y, Onnela JP, Rosenquist JN. Understanding the Demographics of Twitter Users. Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media. 2011.
  27. [44]Chang J, Rosenn I, Backstrom L, Marlow C. ePluribus: Ethnicity on Social Networks. Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media. 2010.
  28. [45]Pennacchiotti M, Popescu A. A machine learning approach to twitter user classification. Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media. 2011.
  29. [46]Zamal FA, Liu W, Ruths D. Homophily and latent attribute inference: Inferring latent attributes of twitter users from neighbors. Proceedings of the Sixth International AAAI Conference on Weblogs and Social Media. 2012.
  30. [47]Kosinski M, Stillwell D, Graepel T. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences 2013; 110 15: 5802– 5805. PMC3625324 23479631
  31. [48]Lampos V, Preotiuc-Pietro D, Cohn T. A user-centric model of voting intention from Social Media. Proc 51st Annual Meeting of the Association for Computational Linguistics. 2013. 993–1003.
  32. [49]Borondo J, Morales AJ, Losada JC, Benito RM. Characterizing and modeling an electoral campaign in the context of twitter: 2011 Spanish presidential election as a case study. Chaos: an interdisciplinary journal of nonlinear science. 2012; 22 2: 023138. 22757545
  33. [50]Mustafaraj E, Finn S, Whitlock C, Metaxas PT. Vocal minority versus silent majority: Discovering the opinions of the long tail. Privacy, security, risk and trust (passat), 2011 ieee third international conference on and 2011 ieee third international conference on social computing (socialcom. 2011.
  34. [52]Broder A. A taxonomy of web search. ACM Sigir forum. 2002; 36 2: 3– 10.
  35. [53]Li X, Wang Y, Acero A. Learning query intent from regularized click graphs. proceedings of the 31st annual international acm sigir conference on research and development in information retrieval. 2008. 339–346.
  36. [54]Java A, Song X, Finin T, Tseng B. Why we twitter: understanding microblogging usage and communities. Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 workshop on Web mining and social network analysis 2007.
  37. [55]Diaz F. Integration of news content into web results. Proceedings of the Second ACM International Conference on Web Search and Data Mining 2009. 182–191.
  38. [57]Huberty ME. Multi-cycle forecasting of congressional elections with social media. Proceedings of the 2nd workshop on Politics. 2013. 23–30.
  39. [58]Wang W, Rothschild D, Goel S, Andrew G. Forecasting Elections with Non-Representative Polls. International Journal of Forecasting. 2015.
  40. [59]Kairam SR, Morris MR, Teevan J, Liebling D, Dumais S. Towards supporting search over trending events with social media. Proceedings of ICWSM 2013, the 7th International AAAI Conference on Weblogs and Social Media. 2013.
  41. [60]Alfonso O, Ke Q, Khandelwal K, Vadrevu S. Exploiting entities in social media. Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval (ESAIR '13 ACM). 2013. 9–12.
  42. [61]Yom-Tov E, Diaz F. The effect of social and physical detachment on information need. ACM Transactions on Information Systems (TOIS). 2013; 31 1: 4:
  43. [62]Wu S, Hofman JM, Mason WA, Watts DJ. Who says what to whom on twitter. Proceedings of the 20th international conference on World wide web 2011.
  44. [63]Morstatter F, Pfeffer J, Liu H, Carle KM. Is the sample good enough? Comparing data from Twitter’s streaming API with Twitter’s firehose. Proceedings of ICWSM. 2013.
  45. [64]Goel S, Hofman JM, Lahaie S, Pennock DM, Watts DJ. Predicting consumer behavior with Web search. Proceedings of the National Academy of Sciences. 2010; 10(741): 17486– 17490. PMC2955127 20876140
  46. [66]Pew Research Political Polarization in the American Public. Pew Research Center. 2014. Available: http://www.people-press.org/2014/06/12/section-5-political-engagement-and-activism.
  47. [67]Butler D. When Google got flu wrong. Nature. 2013; 494(7436), 155: 10.1038/494155a 23407515 · DOI
  48. [68]Broniatowski DA, Paul M, Dredze M. National and Local Influenza Surveillance through Twitter: An Analysis of the 2012–2013 Influenza Epidemic. PLoS ONE 2013; 8(12): e83672: 10.1371/journal.pone.0083672 24349542 PMC3857320 · DOI
  49. [69]Yeager DS, Krosnick JA, Chang L, Javitz HS, Levendusky MS, Simpser A, et al. Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly. 2011.
  50. [72]Kıcıman E. OMG, I have to tweet that! A study of factors that influence tweet rates. Sixth International AAAI Conference on Weblogs and Social Media. 2012.
  51. [73]Hecht B, Hong L, Suh B, Chi EH. Tweets from Justin Bieber's heart: the dynamics of the location field in user profiles. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011. 237–246.

Authors
Diaz, F., Gamon, M., Hofman, J. M., Kıcıman, E., & Rothschild, D.
Affiliations
  • Microsoft Research New York City (United States) (US)
  • Microsoft (United States) (US)
Corresponding author
Rothschild, David
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2016-01-05
Volume
11
Issue
1
Pages
e0145406
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0145406
PMC
PMC4711590
OpenAlex
W2231433198
Cited by
131
References
74
Citations by year
2026: 3 · 2025: 5 · 2024: 12 · 2023: 6 · 2022: 17 · 2021: 14 · 2020: 25 · 2019: 16 · 2018: 20 · 2017: 9 · 2016: 3 · 2015: 1
FWCI
45.21
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
Funders
American Political Science Association
Topics
  • Social Sciences › Communication › Social Media and Politics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Keywords
social media data, Twitter, online surveys, panel survey, 2012 elections, political attitudes, search history, non-stationarity, user participation, sentiment analysis, public opinion measurement
MeSH terms
Humans, Politics, Public Opinion, Surveys and Questionnaires, United States, Reproducibility of Results, Internet, Information Dissemination, Social Media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review