The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Structural Topic Models for Open‐Ended Survey Responses
Abstract
Collection and especially analysis of open‐ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semiautomated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author's gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open‐ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Data clustering: 50 years beyond K-means
Jain, A. K. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2009.09.011 · Cited by 9,368
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 2×
“…recent efforts in digitalization of text, researchers are rapidly reinventing the evidence base of the social sciences by introducing new measures of political and social behavior constructed from large text archives (Grimmer and Stewart 2013; Hopkins and King 2012; King 2009; Lazer et al. 2009).”
BackgroundSelf-citationContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
(2009) · Cited by 1,900
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852
Questions and answers in attitude surveys: Experiments on question form, wording, and context
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90054-w · Cited by 1,713
Inter-Coder Agreement for Computational Linguistics
Artstein, R., & Poesio, M. (2008) · Computational Linguistics · doi:10.1162/coli.07-034-r2 · Cited by 1,665
Optimizing Semantic Coherence in Topic Models
Mimno, D., Wallach, H., Talley, E. M., Leenders, M., & McCallum, A. (2011) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 1,570
Collaborative topic modeling for recommending scientific articles
Wang, C., & Blei, D. M. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020480 · Cited by 1,527
Extracting Policy Positions from Political Texts Using Words as Data
LAVER, M., BENOIT, K., & GARRY, J. (2003) · American Political Science Review · doi:10.1017/s0003055403000698 · Cited by 1,377
Spontaneous giving and calculated greed
Rand, D. G., Greene, J. D., & Nowak, M. A. (2012) · Nature · doi:10.1038/nature11467 · Cited by 1,334
Self-citationAn Introduction to Information Retrieval
Ceri, S., Bozzon, A., Brambilla, M., Della Valle, E., Fraternali, P., & Quarteroni, S. (2013) · Web Information Retrieval · doi:10.1007/978-3-642-39314-3_1 · Cited by 853
Automatic evaluation of topic coherence
Newman, D. J., Lau, J. H., Grieser, K., & Baldwin, T. (2010) · Cited by 846
A Method of Automated Nonparametric Content Analysis for Social Science
Hopkins, D. J., & King, G. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00428.x · Cited by 765
How to analyze political attention with minimal assumptions and costs
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00427.x · Cited by 671
Grimmer, J. (2010) · Political Analysis · doi:10.1093/pan/mpp034 · Cited by 550
Framing Responsibility for Political Issues
IYENGAR, S. (1996) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716296546001006 · Cited by 515
Issue Salience and Party Choice
RePass, D. E. (1929) · American Political Science Review · doi:10.2307/1954456 · Cited by 450
Social Science Methodology: A Unified Framework
Gerring, J. (2012) · Cited by 404
Social Science Methodology: A Criterial Framework
Gerring, J. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815492 · Cited by 384
Kaplan, R. M. (2010) · Cited by 380
The Growth and Development of Experimental Research in Political Science
DRUCKMAN, J. N., GREEN, D. P., KUKLINSKI, J. H., & LUPIA, A. (2006) · American Political Science Review · doi:10.1017/s0003055406062514 · Cited by 374
Computer-Assisted Text Analysis for Comparative Politics
Lucas, C., Nielsen, R. A., Roberts, M. E., Stewart, B. M., Storer, A., & Tingley, D. (2015) · Political Analysis · doi:10.1093/pan/mpu019 · Cited by 366
Self-citationSparse Additive Generative Models of Text
Eisenstein, J., Ahmed, A. A., & Xing, E. P. (2018) · Figshare · doi:10.1184/r1/6476342 · Cited by 277
Anxiety, Immigration, and the Search for Information
Gadarian, S. K., & Albertson, B. (2014) · Political Psychology · doi:10.1111/pops.12034 · Cited by 231
Self-citationThe structural topic model and applied social sciencePreprint
Roberts, M. E., Stewart, B., Tingley, D. H., & Airoldi, E. M. (n.d.) · RePEc: Research Papers in Economics · Cited by 229
Self-citationA Spectral Algorithm for Latent Dirichlet Allocation
Anandkumar, A., Foster, D. P., Hsu, D., Kakade, S. M., & Liu, Y.-K. (2015) · Algorithmica · doi:10.1007/s00453-014-9909-1 · Cited by 223
Do Open-Ended Questions Measure "Salient" Issues?
Geer, J. G. (1991) · Public Opinion Quarterly · doi:10.1086/269268 · Cited by 205
What Do Open-Ended Questions Measure?
Geer, J. G. (1988) · Public Opinion Quarterly · doi:10.1086/269113 · Cited by 203
The Controversy Over Detailed Interviews-An Offer for Negotiation
Lazarsfeld, P. F. (1944) · Public Opinion Quarterly · doi:10.1086/265666 · Cited by 179
General purpose computer-assisted clustering and conceptualization
Grimmer, J., & King, G. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018067108 · Cited by 179
Summarizing topical content with word frequency and exclusivityPreprint
Bischof, J., & Airoldi, E. M. (2012) · arXiv (Cornell University) · Cited by 172
The Random Probe: A Technique for Evaluating the Validity of Closed Questions
Schuman, H. (1966) · American Sociological Review · doi:10.2307/2090907 · Cited by 138
Kelley, S. (1983) · Princeton University Press eBooks · doi:10.1515/9781400855650 · Cited by 123
An Introduction to Bayesian Inference via Variational Approximations
Grimmer, J. (2010) · Political Analysis · doi:10.1093/pan/mpq027 · Cited by 113
The Discrete Infinite Logistic Normal Distribution
Paisley, J. W., Wang, C., & Blei, D. M. (2012) · Bayesian Analysis · doi:10.1214/12-ba734 · Cited by 69
Dimensional Reduction of Word-Frequency Data as a Substitute for Intersubjective Content Analysis
Simon, A. F., & Xenos, M. A. (2004) · Political Analysis · doi:10.1093/pan/mph004 · Cited by 47
The Future of Political Science
(2009) · Routledge eBooks · doi:10.4324/9780203882313 · Cited by 32
Hopkins, D. J. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2163769 · Cited by 10
Cited by
stm : An R Package for Structural Topic Models
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019) · Journal of Statistical Software · doi:10.18637/jss.v091.i02 · Cited by 1,714
Self-citationThe Diversity–Innovation Paradox in Science
Hofstra, B., Kulkarni, V. V., Munoz-Najar Galvez, S., He, B., Jurafsky, D., & McFarland, D. A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915378117 · Cited by 975
A Model of Text for Experimentation in the Social Sciences
Roberts, M. E., Stewart, B. M., & Airoldi, E. M. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1141684 · Cited by 728
Self-citationTopic Modeling in Management Research: Rendering New Theory from Textual Data
Hannigan, T. R., Haans, R. F. J., Vakili, K., Tchalian, H., Glaser, V. L., Wang, M. S., Kaplan, S., & Jennings, P. D. (2019) · Academy of Management Annals · doi:10.5465/annals.2017.0099 · Cited by 621
Corporate funding and ideological polarization about climate change
Farrell, J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1509433112 · Cited by 603
Metadata
- Authors
- Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G.
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2014-03-06
- Volume
- 58
- Issue
- 4
- Pages
- 1064-1082
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2096974619
- Cited by
- 1,952
- References
- 46
- FWCI
- 230.94
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.