CoChat
Sign Up
PreprintCited by 24Sentiment Analysis and Opinion MiningOpinion Dynamics and Social InfluenceComputational and Text Analysis Methods

Language Models Trained on Media Diets Can Predict Public Opinion

Open access
Chu, Eric, Andreas, Jacob, Ansolabehere, Stephen, Roy, Deb·2023·arXiv
DOI 10.48550/arxiv.2303.16779arXiv 2303.16779
Open this paper in CoChat — free accountOpen PDF

Abstract

Public opinion reflects and shapes societal behavior, but the traditional survey-based tools to measure it are limited. We introduce a novel approach to probe media diet models -- language models adapted to online news, TV broadcast, or radio show content -- that can emulate the opinions of subpopulations that have consumed a set of media. To validate this method, we use as ground truth the opinions expressed in U.S. nationally representative surveys on COVID-19 and consumer confidence. Our studies indicate that this approach is (1) predictive of human judgements found in survey response distributions and robust to phrasing and channels of media exposure, (2) more accurate at modeling people who follow media more closely, and (3) aligned with literature on which types of opinions are affected by media consumption. Probing language models provides a powerful new method for investigating media effects, has practical applications in supplementing polls and forecasting public opinion, and suggests a need for further study of the surprising fidelity with which neural language models can predict human responses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.16779

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Fine-tuning selected models on US partisan Twitter data can be interpreted as exposing the model to a 'news diet' of partisan tweets.

    “This process can be interpreted as exposing the model to a ‘news diet’ of partisan tweets, aligning with the interpretation by ref. 74.”

    From Generative language models exhibit social identity biases (2024) · cites this paper as [74]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Baum, M. A. & Groeling, T. New media and the polarization of american political discourse. Polit. Commun. 25, 345-365, 10.1080/10584600802426965 (2008). https://doi.org/10.1080/10584600802426965. · DOI
  2. Flaxman, S., Goel, S. & Rao, J. M. Filter bubbles, echo chambers, and online news consumption. Public opinion quarterly 80, 298-320 (2016).
  3. Stimson, J. A., MacKuen, M. B. & Erikson, R. S. Dynamic representation. Am. political science review 89, 543-565 (1995).
  4. Ansolabehere, S. & Schaffner, B. F. Taking the study of political behavior online. In The Oxford Handbook of Polling and Survey Methods, 76 (Oxford University Press New York, NY, 2018).
  5. Lippmann, W. Public opinion (Routledge, 2017).
  6. Valkenburg, P. M. & Peter, J. Comm research-views from europe| five challenges for the future of media-effects research. Int. J. Commun. 7, 19 (2013).
  7. Valkenburg, P. M., Peter, J. & Walther, J. B. Media effects: Theory and research. Annu. review psychology 67, 315-338 (2016).
  8. O'Connor, B., Balasubramanyan, R., Routledge, B. & Smith, N. From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the International AAAI Conference on Web and Social Media, vol. 4 (2010).
  9. Grimmer, J. Representational style in Congress: What legislators say and why it matters (Cambridge University Press, 2013).
  10. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186, 10.18653/v1/N19-1423 (Association for Computational Linguistics, Minneapolis, Minnesota, 2019). · DOI · Paper page
  11. Petroni, F. et al. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2463-2473 (2019).
  12. Brown, T. et al. Language models are few-shot learners. Adv. neural information processing systems 33, 1877-1901 (2020).
  13. Chowdhery, A. et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
  14. Bartels, L. M. Messages received: The political impact of media exposure. Am. political science review 267-285 (1993).
  15. Holtzman, A., West, P., Shwartz, V., Choi, Y. & Zettlemoyer, L. Surface form competition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7038-7051 (2021).
  16. Bursztyn, L., Rao, A., Roth, C. P. & Yanagizawa-Drott, D. H. Misinformation during a pandemic. Working Paper 27417, National Bureau of Economic Research (2020). 10.3386/w27417. · DOI
  17. Allcott, H. et al. Polarization and public health: Partisan differences in social distancing during the coronavirus pandemic. Working Paper 26946, National Bureau of Economic Research (2020). 10.3386/w26946. · DOI
  18. Simonov, A., Sacher, S. K., Dubé, J.-P. H. & Biswas, S. The persuasive effect of fox news: Non-compliance with social distancing during the covid-19 pandemic. Working Paper 27237, National Bureau of Economic Research (2020). 10.3386/w27237. · DOI
  19. Hollanders, D. & Vliegenthart, R. The influence of negative newspaper coverage on consumer confidence: The dutch case. J. Econ. Psychol. 32, 367-373, https://doi.org/10.1016/j.joep.2011.01.003 (2011). · DOI
  20. Baum, M. Soft news goes to war: Public opinion and American foreign policy in the new media age (Princeton University Press, 2003).
  21. Koh, P. W. & Liang, P. Understanding black-box predictions via influence functions. In International conference on machine learning, 1885-1894 (PMLR, 2017).
  22. Weidinger, L. et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021). · arXiv
  23. Converse, P. E. The nature of belief systems in mass publics (1964). Critical Rev. 18, 1-74, 10.1080/08913810608443650 (2006). https://doi.org/10.1080/08913810608443650. · DOI
  24. Zaller, J. R. et al. The nature and origins of mass opinion (Cambridge university press, 1992).
  25. Tourangeau, R. & Rasinski, K. A. Cognitive processes underlying context effects in attitude measurement. Psychol. bulletin 103, 299 (1988).
  26. Lazarsfeld, P. F., Berelson, B. & Gaudet, H. The people's choice. In The people's choice (Columbia University Press, 1944).
  27. Schwitzgebel, E. Belief. In The Routledge Companion to Epistemology, 40-50 (Routledge, 2011).
  28. Gauthier, J. & Levy, R. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 529-539 (2019).
  29. SEARS, D. O. & FREEDMAN, J. L. SELECTIVE EXPOSURE TO INFORMATION: A CRITICAL REVIEW*. Public Opin. Q. 31, 194-213, 10.1086/267513 (1967). https://academic.oup.com/poq/article-pdf/31/2/194/5132577/31-2-194.pdf. · DOI
  30. Sunstein, C. R. Republic.Com 2.0 (Princeton University Press, USA, 2007).
  31. Pariser, E. The filter bubble: What the Internet is hiding from you (Penguin UK, 2011).
  32. Thibodeau, P. H. & Boroditsky, L. Metaphors we think with: The role of metaphor in reasoning. PloS one 6, e16782 (2011).
  33. Tan, C., Lee, L. & Pang, B. The effect of wording on message propagation: Topic-and author-controlled natural experiments on twitter. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 175-185 (2014).
  34. Pfeiffer, J. et al. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 46-54 (2020).
  35. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. & Dean, J. Distributed representations of words and phrases and their compositionality. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z. & Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems, vol. 26 (Curran Associates, Inc., 2013).
  36. Edunov, S., Ott, M., Auli, M. & Grangier, D. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 489-500 (2018).

Cited by

Sorted by citations

Authors
Chu, E., Andreas, J., Ansolabehere, S., & Roy, D.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Harvard University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2303.16779
arXiv
2303.16779
OpenAlex
W4361807105
Cited by
24
References
0
Citations by year
2026: 1 · 2025: 6 · 2024: 8 · 2023: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Opinion Dynamics and Social Influence
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
public opinion prediction, language models, media influence, media use, consumer confidence, neural language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review