Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale
Open accessAbstract
Can artificial intelligence accurately label open-text survey responses? We compare the accuracy of six large language models (LLMs) using a few-shot approach, three supervised learning algorithms (SVM, DistilRoBERTa, and a neural network trained on BERT embeddings), and a second human coder on the task of categorizing “most important issue” responses from the British Election Study Internet Panel into 50 categories. For the scenario where a researcher lacks existing training data, the accuracy of the highest-performing LLM (Claude-1.3: 93.9%) neared human performance (94.7%) and exceeded the highest-performing supervised approach trained on 1000 randomly sampled cases (neural network: 93.5%). In a scenario where previous data has been labeled but a researcher wants to label novel text, the best LLM’s (Claude-1.3: 80.9%) few-shot performance is only slightly behind the human (88.6%) and exceeds the best supervised model trained on 576,000 cases (DistilRoBERTa: 77.8%). PaLM-2, Llama-2, and the SVM all performed substantially worse than the best LLMs and supervised models across all metrics and scenarios. Our results suggest that LLMs may allow for greater use of open-ended survey questions in the future.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several prominent LLMs produced the same classification as highly trained coders analyzing statements about British Elections roughly 95% of the time.
“They find LLMs produced the same classification roughly 95% of the time.”
From Can Generative AI improve social science? (2024) · cites this paper as [52]
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Structural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B., Tingley, D. H., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B. L., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 1,969 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Yet arti fi cial intelligence ’ s capability has skyrocketed in recent years (Kaplan et al., 2020).”
BackgroundHuang, F., Kwak, H., & An, J. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587368 · Cited by 247 · see citing works
Lai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233 · see citing works
A Review of Public Issue Salience: Concepts, Determinants and Effects on Voting
Dennison, J. (2019) · Political Studies Review · doi:10.1177/1478929918819264 · Cited by 179 · see citing works
RTextTools: A Supervised Learning Package for Text Classification
Timothy, P. J., Collingwood, L., Amber, E. B., et al. (2013) · The R Journal · doi:10.32614/rj-2013-001 · Cited by 121 · see citing works
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104 · see citing works
Schuldt, J. P., & Roh, S. (2014) · Environmental Communication · doi:10.1080/17524032.2014.909510 · Cited by 84 · see citing works
Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPTPreprint
Amin, M. M., Cambria, E., & Schuller, B. W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.03186 · Cited by 33 · see citing works
An analysis of the public's personal, national and EU issue priorities
Bevan, S., Jennings, W., & Wlezien, C. B. (2015) · Journal of European Public Policy · doi:10.1080/13501763.2015.1070191 · Cited by 17 · see citing works
Sentiment Analysis of Russia-Ukraine Conflict Tweets Using RoBERTa
Ramos, L., & Chang, O. (2023) · Uniciencia · doi:10.15359/ru.37-1.23 · Cited by 17 · see citing works
Domain-Specific Language Model Pre-Training for Korean Tax Law Classification
Gu, Y. H., Piao, X., Yin, H., Dong, J., Zheng, R., & Yoo, S. J. (2022) · IEEE Access · doi:10.1109/access.2022.3164098 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47 · see citing works
- Authors
- Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., & Schmedeman, P.
- Affiliations
- United States Military Academy (US)
- University of Manchester (GB)
- University of Bristol (GB)
- Corresponding author
- Mellon, Jonathan
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2024-01
- Volume
- 11
- Issue
- 1
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W4391758695
- Cited by
- 61
- References
- 15
- Citations by year
- 2026: 15 · 2025: 37 · 2024: 8 · 2023: 1
- FWCI
- 52.24
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Research & Politics (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- Bristol Research (University of Bristol) (publishedVersion, cc-by)
- ORCA Online Research @Cardiff (Cardiff University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Explore Bristol Research (submittedVersion, cc-by)
- Funders
- Economic and Social Research Council (ES/S012435/1, ES/S015671/1)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- large language models, open-ended survey responses, British Election Study, few-shot learning, supervised learning comparison, text classification, political attitudes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
