The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Structural Topic Models for Open‐Ended Survey Responses
Abstract
Collection and especially analysis of open‐ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semiautomated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author's gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open‐ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Data clustering: 50 years beyond K-means
Jain, A. K. (2010) · Pattern Recognition Letters · doi:10.1016/j.patrec.2009.09.011 · Cited by 9,368
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 2×
“…recent efforts in digitalization of text, researchers are rapidly reinventing the evidence base of the social sciences by introducing new measures of political and social behavior constructed from large text archives (Grimmer and Stewart 2013; Hopkins and King 2012; King 2009; Lazer et al. 2009).”
BackgroundSelf-citationContent Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00826.x · Cited by 3,068
(2009) · Cited by 1,900
Cited by
stm : An R Package for Structural Topic Models
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019) · Journal of Statistical Software · doi:10.18637/jss.v091.i02 · Cited by 1,714
Self-citationThe Diversity–Innovation Paradox in Science
Hofstra, B., Kulkarni, V. V., Munoz-Najar Galvez, S., He, B., Jurafsky, D., & McFarland, D. A. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915378117 · Cited by 975
A Model of Text for Experimentation in the Social Sciences
Roberts, M. E., Stewart, B. M., & Airoldi, E. M. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1141684 · Cited by 728
Self-citationTopic Modeling in Management Research: Rendering New Theory from Textual Data
Hannigan, T. R., Haans, R. F. J., Vakili, K., Tchalian, H., Glaser, V. L., Wang, M. S., Kaplan, S., & Jennings, P. D. (2019) · Academy of Management Annals · doi:10.5465/annals.2017.0099 · Cited by 621
Corporate funding and ideological polarization about climate change
Farrell, J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1509433112 · Cited by 603
Stier, S., Bleier, A., Lietz, H., & Strohmaier, M. (2018) · Political Communication · doi:10.1080/10584609.2017.1334728 · Cited by 566
Quantitative analysis of large amounts of journalistic texts using topic modelling
Jacobi, C., van Atteveldt, W., & Welbers, K. (2016) · Digital Journalism · doi:10.1080/21670811.2015.1093271 · Cited by 524
Prescriptive analytics: Literature review and research challenges
Lepenioti, K., Bousdekis, A., Apostolou, D., & Mentzas, G. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.04.003 · Cited by 501
“…making problems in different contexts are: Artificial Neural Networks (Domingos, Burguillo, & Lenaerts, 2017), Bayesian Networks (Said, Shahzad, Zamai, Hubac, & Tollenaere, 2016), topic models (Roberts et al., 2014), fuzzy sets (Pal & Ceglarek, 2013), Markov models (Bousdekis et al., 2018).”
Denny, M. J., & Spirling, A. (2018) · Political Analysis · doi:10.1017/pan.2017.44 · Cited by 463
Chen, X., Zou, D., Cheng, G., & Xie, H. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103855 · Cited by 403
Rand, D. G. (2016) · Psychological Science · doi:10.1177/0956797616654455 · Cited by 395
Self-citationData integration enables global biodiversity synthesis
Heberling, J. M., Miller, J. T., Noesgaard, D., Weingart, S. B., & Schigel, D. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2018093118 · Cited by 380
What do hotel customers complain about? Text analysis using structural topic model
Hu, N., Zhang, T., Gao, B., & Bose, I. (2019) · Tourism Management · doi:10.1016/j.tourman.2019.01.002 · Cited by 376
Computer-Assisted Text Analysis for Comparative Politics
Lucas, C., Nielsen, R. A., Roberts, M. E., Stewart, B. M., Storer, A., & Tingley, D. (2015) · Political Analysis · doi:10.1093/pan/mpu019 · Cited by 366
Self-citationMachine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Self-citationLarge-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300
Beliefs about Climate Beliefs: The Importance of Second-Order Opinions for Climate Politics
Mildenberger, M., & Tingley, D. (2019) · British Journal of Political Science · doi:10.1017/s0007123417000321 · Cited by 300
Self-citationWelbers, K., Van Atteveldt, W., & Benoit, K. (2017) · Communication Methods and Measures · doi:10.1080/19312458.2017.1387238 · Cited by 276
Text Mining in Organizational Research
Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismihók, G., & Den Hartog, D. N. (2018) · Organizational Research Methods · doi:10.1177/1094428117722619 · Cited by 275
Bauer, P. C., Barberá, P., Ackermann, K., & Venetz, A. (2017) · Political Behavior · doi:10.1007/s11109-016-9368-2 · Cited by 269
Text-mining the signals of climate change doubt
Boussalis, C., & Coan, T. G. (2016) · Global Environmental Change · doi:10.1016/j.gloenvcha.2015.12.001 · Cited by 263
Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research
Rodriguez, P. L., & Spirling, A. (2022) · The Journal of Politics · doi:10.1086/715162 · Cited by 254
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
Self-citationAsh, E., & Hansen, S. (2023) · Annual Review of Economics · doi:10.1146/annurev-economics-082222-074352 · Cited by 197
Cognitive reflection correlates with behavior on Twitter
Mosleh, M., Pennycook, G., Arechar, A. A., & Rand, D. G. (2021) · Nature Communications · doi:10.1038/s41467-020-20043-0 · Cited by 175
Self-citationRogers, N. T., Waterlow, N. R., Brindle, H., Enria, L., Eggo, R. M., Lees, S., & Roberts, C. H. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.575091 · Cited by 140
Sustainable finance literacy and the determinants of sustainable investing
Filippini, M., Leippold, M., & Wekhof, T. (2024) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2024.107167 · Cited by 144
Fischer-Preßler, D., Schwemmer, C., & Fischbach, K. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.05.012 · Cited by 133
Concealing Corruption: How Chinese Officials Distort Upward Reporting of Online Grievances
PAN, J., & CHEN, K. (2018) · American Political Science Review · doi:10.1017/s0003055418000205 · Cited by 125
Russian Twitter accounts and the partisan polarization of vaccine discourse, 2015-2017
Walter, D., Ophir, Y., & Jamieson, K. H. (2020) · American Journal of Public Health · doi:10.2105/ajph.2019.305564 · Cited by 111
Employees’ perceptions of relational communication in full-time remote work in the public sector
Jämsen, R., Sivunen, A., & Blomqvist, K. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107240 · Cited by 98
The Anchoring Effect in Decision-Making with Visual Analytics
Cho, I., Wesslen, R., Karduni, A., Santhanam, S., Shaikh, S., & Dou, W. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585665 · Cited by 88
Tvinnereim, E., Liu, X., & Jamelske, E. M. (2017) · Climatic Change · doi:10.1007/s10584-016-1871-2 · Cited by 74
Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., & Schmedeman, P. (2024) · Research & Politics · doi:10.1177/20531680241231468 · Cited by 61
Planning prompts increase and forecast course completion in massive open online courses
Yeomans, M., & Reich, J. (2017) · Proceedings of the Seventh International Learning Analytics & Knowledge Conference · doi:10.1145/3027385.3027416 · Cited by 61
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58
Mumtaz, S., Carmichael, J., Weiss, M., & Nimon-Peters, A. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13099-8 · Cited by 55
Baburajan, V., e Silva, J. D. A., & Pereira, F. C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3040904 · Cited by 53
Skogstad, G., & Wilder, M. (2019) · Policy Sciences · doi:10.1007/s11077-019-09351-5 · Cited by 41
Yuan, S., Chen, Y., Vojta, S., & Chen, Y. (2024) · New Media & Society · doi:10.1177/14614448221122202 · Cited by 23
Utilization and challenges of artificial intelligence in the energy sector
Park, C., & Kim, M. (2026) · Energy & Environment · doi:10.1177/0958305x241258795 · Cited by 15
Mourtgos, S. M., Adams, I. T., McCrain, J., Schiff, K. J., & Schiff, D. S. (2024) · Criminology & Public Policy · doi:10.1111/1745-9133.12674 · Cited by 8
Assessing and dissociating virtues from the ‘bottom up’: A case study of generosity vs. fairness
Kraft-Todd, G. T., Kleiman-Weiner, M., & Young, L. (2023) · The Journal of Positive Psychology · doi:10.1080/17439760.2022.2154254 · Cited by 7
Metadata
- Authors
- Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G.
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2014-03-06
- Volume
- 58
- Issue
- 4
- Pages
- 1064-1082
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W2096974619
- Cited by
- 1,952
- References
- 46
- FWCI
- 230.94
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.