Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
We identify three gaps that limit the utility and obstruct the progress of computational text analysis methods (CTAM) for social science research. First, we contend that CTAM development has prioritized technological over validity concerns, giving limited attention to the operationalization of social scientific measurements. Second, we identify a mismatch between CTAMs’ focus on extracting specific contents and document-level patterns, and social science researchers’ need for measuring multiple, often complex contents in the text. Third, we argue that the dominance of English language tools depresses comparative research and inclusivity toward scholarly communities examining languages other than English. We substantiate our claims by drawing upon a broad review of methodological work in the computational social sciences, as well as an inventory of leading research publications using quantitative textual analysis. Subsequently, we discuss implications of these three gaps for social scientists’ uneven uptake of CTAM, as well as the field of computational social science text research as a whole. Finally, we propose a research agenda intended to bridge the identified gaps and improve the validity, utility, and inclusiveness of CTAM.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 1×
“…absence of operational considerations in the development and documentation of CTAM, social scientists trying to make an informed choice about their use of computational methods frequently fail to find relevant guidance in the existing methodological literature (Grimmer & Stewart, 2013).”
BackgroundBlei, D. M., & Lafferty, J. D. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143859 · Cited by 3,173
stm : An R Package for Structural Topic Models
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019) · Journal of Statistical Software · doi:10.18637/jss.v091.i02 · Cited by 1,714
Sentiment strength detection in short informal text
Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21416 · Cited by 1,449
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · cited 2×
“Yet, many available computational methods are conceptualized and built as standalone tools, using different coding languages and data standards and offering specialized interfaces that do not support concatenation within more complex pipelines (Liu & Zhang, 2012; Tenney et al., 2019).”
BackgroundMethodPapacharissi, Z., & de Fatima Oliveira, M. (2008) · The International Journal of Press/Politics · doi:10.1177/1940161207312676 · Cited by 251
Metadata
- Authors
- Baden, C., Pipal, C., Schoonvelde, M., & van der Velden, M. A. C. G.
- Venue
- Communication Methods and Measures
- Publisher
- Informa UK Limited
- Published
- 2021-12-27
- Volume
- 16
- Issue
- 1
- Pages
- 1-18
- ISSN
- 1931-2458 (ISSN-L)
- 1931-2466
- OpenAlex
- W4200425490
- Cited by
- 182
- References
- 8
- FWCI
- 74.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda. https://app.cochat.ai/papers/10.1080/19312458.2021.2015574
- The paper
- Baden, C., Pipal, C., Schoonvelde, M., & van der Velden, M. A. C. G. (2022). Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda. Communication Methods and Measures. https://doi.org/10.1080/19312458.2021.2015574
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.