CoChat
Sign Up
Cited by 3,308Computational and Text Analysis MethodsSentiment Analysis and Opinion MiningDigital Humanities and Scholarship

The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.

Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts

Open access
Grimmer, Justin, Stewart, Brandon M.·2013·Political Analysis
DOI 10.1093/pan/mps028
Open this paper in CoChat — free accountOpen PDF

Abstract

We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.

Key findings

  • Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
  • In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
  • In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
  • In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.

Methods

  • Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
  • Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
  • Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.

Limitations

  • The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
  • The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
  • Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/pan/mps028

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

12 of 89 cited sources checked · 12 supported · 0 flagged

Show all 121 claims

  • Probably not a substantial claim. 158
  • It may state two separate claims as one. 34
174 of 174

  • The authors state this as an assumption.

    Documents are a bag of words, where word order does not inform analyses.

    “We will assume documents are a bag of words, where order does not inform our analyses.”

  • The authors state this as an assumption.

    Ideology dominates the language used in the text.

    “both methods rely heavily on the assumption that ideology dominates the language used in the text.”

  • The paper relies on this without support.

    For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.

    “For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”

  • The paper relies on this without support.

    Measures from dictionaries are rarely validated.

    “But measures from dictionaries are rarely validated.”

  • The paper relies on this without support.

    Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.

    “Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”

  • The authors state this as an assumption.

    Given a document's category, the words are generated independently.

    “the model assumes that, given a document's category, the words are generated independently”

  • The authors state this as an assumption.

    There is some unobserved function that describes the relationship between the words and the labels.

    “Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”

  • The authors state this as an assumption.

    The conditional distributions are identical in the training and test sets.

    “But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”

  • The authors state this as an assumption.

    LDA assumes that each document is a mixture of topics.

    “LDA assumes that each document is a mixture of topics.”

  • The authors state this as an assumption.

    LDA assumes that a two-step process generates each word.

    “LDA assumes that a two-step process generates each word.”

  • The authors state this as an assumption.

    The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.

    “The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”

  • The authors state this as an assumption.

    Each document's proportions are drawn from a common Dirichlet prior.

    “We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”

  • The authors state this as an assumption.

    Actors' ideological leanings determine what is discussed in texts.

    “Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”

  • The authors state this as an assumption.

    Politicians reside in a low-dimensional political space.

    “Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”

  • The authors state this as an assumption.

    A politician's or party's position in the political space affects the rate at which words are used in texts.

    “A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”

  • The authors state this as an assumption.

    Each word j from individual i is drawn from a Poisson distribution with rate ij.

    “Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”

  • The authors state this as an assumption.

    The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.

    “ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”

  • The paper relies on this without support.

    Text analysis methods are necessarily incorrect models of language.

    “And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”

  • The paper relies on this without support.

    The Senate is now polarized.

    “a necessity for any valid scaling in the now polarized Senate.”
19 of 19

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Naïve Bayes is said to assign documents to other documents

    Minor

    The description says the classifier assigns each test document to another document, whereas the paper defines its task as assigning documents to categories. The surrounding explanation indicates that "document" is an error here.

1 of 1

Citations

Cites

Flagged first, then by citations

Show all 109 references

4 of the 109 references couldn’t be matched to a known paper.

Research works printed but not linked: 12

  1. Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
  2. Hart, 2000, Diction 5.0: The text analysis program
  3. Greene, 2007, Econometric analysis
  4. Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
  5. van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
  6. ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
  7. Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
  8. Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
  9. Jackman, 2006, Data from Web into R, The Political Methodologist, 14
  10. Fenno, 1978, Home style: House members in their districts
  11. Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
  12. Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference

Cited by

Sorted by citations

Items 1,201–1,300 of 3,308

Related

See all

Metadata

Authors
Grimmer, J., & Stewart, B. M.
Affiliations
  • Stanford University (US)
  • Harvard University (US)
Corresponding author
Grimmer, Justin
ORCID
Venue
Political Analysis
Publisher
Cambridge University Press (CUP)
Published
2013
Volume
21
Issue
3
Pages
267-297
ISSN
  • 1047-1987 (ISSN-L)
  • 1476-4989
DOI
10.1093/pan/mps028
OpenAlex
W2095655043
Cited by
3,297
References
109
Citations by year
2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
FWCI
537.75
Type
journal article
Language
en
Open access
Licence
www.cambridge.org/core/terms (unspecified, 1464 d)
Locations
  • Political Analysis (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • National Science Foundation
  • University of Cambridge
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
Keywords
computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review