The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
161 Own interpretation · 38 No basis given · 28 Sources not found · 26 No full text of the sources · 21 Own result · 11 Supported · 8 Not checked yet · 1 Partly supported · 1 Own method
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Supportedcites Hand 2006 source
Section 3, “Design Sample Selection,” says the classical supervised-classification paradigm assumes the design set is randomly drawn from the same distribution as future points to be classified. That supports the citing sentence’s application of the assumption to documents; the supplied copy does not provide a page number. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract says that determining in advance which clustering objective function will produce a useful or insightful partition for a given dataset is unknown and difficult, if not logically impossible. It also says there is no method for choosing beforehand which unsupervised approach will yield the most useful clustering for a particular application. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
In Model Output (page unavailable), the paper calls its speech-topic assignments “parameters of direct interest” and uses them to measure attention to political topics over time and by speaker group. Model Specification and Sensitivity Analysis (page unavailable) says the authors selected topic categories to correspond to areas of governmental competence. Together, these support the paraphrase. (LLM, conf=0.87)
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
The model description explains that speech topics are inferred and their probabilities vary by day. The Abstract reports a daily series of attention to topics in U.S. Senate speeches. These passages support the quoted claim; they do not establish the separate 'first topic model' characterization in the citing sentence. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology (§2.4) says the authors project a metric space of clusterings into two Euclidean dimensions and that nearby points represent similar clusterings. Methodology (§2.3) defines similarity by how documents are grouped across clusterings, matching the citing sentence’s paraphrase. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Categorizing Texts: Methods, Assumptions, and Costs” (page not available), the paper says its topic model infers categories rather than requiring them to be fixed beforehand, and shifts effort to postanalysis. “Topic Interpretation and Intratopic Semantic Validity” (page not available) describes examining the resulting topics and documents to assess their validity. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“empirically these sentences are rare.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the restrained category, the recall is 0.75.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 158
- It may state two separate claims as one. 34
Every source the paper cites for this claim was checked and supports it.
In the paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Supportedcites Berinsky, Huber, and Lenz 2012 source
The Introduction says MTurk can facilitate low-cost experiments, and Section 2 describes how researchers post tasks for workers and recruit survey participants. The Abstract also describes MTurk experiments as easy to field. Together, these passages support the claim that the platform can make data acquisition efficient. (LLM, conf=0.91)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Supportedcites Monroe, Colaresi, and Quinn (2008) source
Section 3 describes methods for selecting and evaluating words that distinguish Democratic from Republican Senate speeches. Section 3.1 makes clear that the categories are already identified: the speakers’ partisanship is observed, not something the methods need to infer. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Grimmer and King 2011 source
Section 3.1 (page unavailable) defines cluster quality from human judgments as within-cluster similarity minus between-cluster similarity, and Section 3 reports using that measure to compare clusterings. The measure is comparative rather than an absolute test of semantic validity, consistent with the paper’s caution that there is no reliable gold-standard clustering for this purpose. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract (p. 2643) says the authors develop a space containing existing clustering methods and millions of additional solutions based on combinations of existing clusterings, then enable users to explore it quickly. Section 2, Methodology, describes how users navigate that space to select useful clusterings. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology section describes CAC as organizing a large number of possible clusterings in a visual space so researchers can efficiently select useful ones. The Abstract likewise says users can explore a space containing existing methods and millions of additional solutions and quickly identify useful clusterings. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Exogenous Events and Predictive Validity,” the paper defines predictive validity as correspondence between a measure and external events, then links spikes in topic attention to events including 9/11 and debates over military action. The Discussion says examining relationships with such events helps assess the validity of its attention measures. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The topics in Table 4 exemplify the utility of unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language is the medium for politics and political conflict.”
- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Politics and political conflict often occur in the written and spoken word.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential when applying automated content methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it substantially reduces the costs of analyzing large collections of text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“to understand what politics is about we need to know what political actors are saying and writing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“require extensive and problem-specific validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they are no substitute for careful thought and close reading”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of any one classifier can vary substantially across contexts”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“And hiring coders to manually read all documents is still very expensive.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One way to improve upon dictionaries are supervised methods”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“automated content methods are incorrect models of language.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding the separating words is also relatively easy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tone is just one type of analysis a dictionary method can perform.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is often difficult, and sometimes impossible, to validate the output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe described above is one way to represent text as data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather than replace humans, computers amplify human abilities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is also substantial variation within families of models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validations are necessary to establish its meaning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- The authors state this as an assumption: Ideology dominates the language used in the text.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The data generation process for any text (including this one) is a mystery-even to linguists.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of applied research uses stemming.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“sometimes the problem calls for a completely different approach.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stemmer is a much cruder algorithm, but considerably faster.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance does not seem to matter for most applications”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“automated scraping methods make acquiring these data easier (Jackman 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These programs simply provide the researcher with output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying FAC algorithms to texts may have mixed results”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
What the sources say
- Not checked — no full text of the sourcecites Spirling (2012) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Eggers and Hainmueller 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated methods also rely on having a sufficient number of words to use reliably.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Porter 1980 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Not matchedcites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these require the close reading of texts and thoughtful analysis by the researcher.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assigning texts to categories is the most common use of content analysis methods in political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“not all of language's complexity is necessary to effectively analyze texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
What the sources say
- Not matchedcites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.3
What the sources say
- Not checked — no full text of the sourcecites van der Vaart, Dudoit, and van der Laan 2006 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Not matchedcites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
- Not checked — no full text of the sourcecites Hastie, Tibshirani, and Friedman 2001 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Dictionary methods require scholars to identify words that separate classes beforehand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number necessary will depend upon the specific application of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
- Not checked — no full text of the sourcecites Hillard, Purpura, and Wilkerson 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAC methods are useful, but are necessarily model dependent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1 FAC
What the sources say
- Not matchedcites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- The authors state this as an assumption: Given a document's category, the words are generated independently.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Supervised and dictionary methods assume a well-defined set of categories.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Yiannakis 1982 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the K-means' objective function is multimodal and non-continuous.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not checked — no full text of the sourcecites Blei 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and therefore several random restarts are often necessary to find the global optimum.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § 6.1.2 Mixed membership models · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The goal is revelation of substantively interesting information.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To estimate a topic, the models use the co-occurrence of words across documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Situating the models in a data-generating process makes including additional structure straightforward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preprocessed texts represent a substantial simplification of the documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CAC methods provide a way to explore these different assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, CAC provides an accelerated way to explore new collections of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many other distance metrics are possible, and each will lead to different clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But CAC does place substantial burden on users to identify clusterings that are useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.2 Unsupervised Methods for Scaling
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Lowe 2008 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But nearly all political actors speak.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We caution that this validation does not obviate the need for unsupervised methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that any one method's performance will be context specific.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimating locations using existing data is often difficult and sometimes impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not matchedcites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Experimental — accuracy not yet measuredIn the citing paper
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
In the citing paper
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 25,941
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
(1967) · Cited by 23,719
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Items 1,801–1,900 of 3,308
Towards AI-enabled evidence-based policymaking in science and technology: a conceptual framework
Namdarian, L. (2025) · Technology Analysis and Strategic Management · doi:10.1080/09537325.2025.2486003 · Cited by 3
“…studies have emphasised the application of automated content analysis and natural language processing (NLP) in identifying policymaking patterns (Grimmer and Stewart 2013; Jin and Mihalcea 2022), while others have explored the role of data mining and AI in facilitating policy-makers ’…”
Zhang, M., Kober, T., Patt, A., & Panos, E. (2025) · Energy Strategy Reviews · doi:10.1016/j.esr.2025.101830 · Cited by 3
Probabilistic Record Linkage Using Pretrained Text Embeddings
Ornstein, J. T. (2025) · Political Analysis · doi:10.1017/pan.2025.10016 · Cited by 3
“…be pre-processed by stemming, converting to lowercase, or any other steps that one might take to reduce complexity in a bag-of-words representation (Grimmer and Stewart 2013); performance will generally be improved if we embed text as it is most likely to appear in the training corpus (e.g.,…”
Detecting Group Mentions in Political Rhetoric A Supervised Learning Approach
Licht, H., & Sczepanski, R. (2025) · British Journal of Political Science · doi:10.1017/s0007123424000954 · Cited by 11
“As in other applications (cf. Grimmer and Stewart 2013; Quinn et al. 2010), this approach can be considered the most valid compared to semi-or fully automated methods.”
Candidate Positions, Responsiveness, and Returns to Extremism
Meisels, M. (2025) · The Journal of Politics · doi:10.1086/738505 · Cited by 4
AI in innovation research: an overview of transformers
Mastrogiorgio, M. (2025) · Industry and Innovation · doi:10.1080/13662716.2025.2562539 · Cited by 3
“In recent years, text data have experienced a surge in popularity in the social sciences (Ash and Hansen 2023; Gentzkow, Kelly, and Taddy 2019; Grimmer and Stewart 2013; Grimmer, Roberts, and Stewart 2022).”
Measuring inferred responsible leadership intentions: formative index development
Ünür, M., Karakaş, H., Derdowski, L. A., Linge, T. T., & Araslı, H. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02397-z · Cited by 3
Recoding Reality: A Case Study of YouTube Reactions to Generative AI Videos
Çallı, L., & Çallı, B. A. (2025) · Systems · doi:10.3390/systems13100925 · Cited by 5
Scrolling Through Chaos: The Implications of TikTok for Crisis Sensemaking
Khoury, C., & Hemsley, J. (2025) · Social Media + Society · doi:10.1177/20563051251382470 · Cited by 3
“Comments were classified using an adapted version of Heverin and Zach’s (2012) collective sensemaking codebook, with classification performed through the Bidirectional Encoder Representations from Transformers (BERT) language model (Grimmer & Stewart, 2013).”
Kurniati, Y., & Abdillah, A. (2025) · Laws · doi:10.3390/laws14060079 · Cited by 14
“Comprehensive study analysis and content analysis methods (Krippendorff 2018; Grimmer and Stewart 2013) are used in this study, where this method compares the legal systems of various existing regulations from two countries chosen for their uniqueness in the case of immigrant workers, namely…”
Ye, M., & Guo, R. (2025) · Frontiers in Marine Science · doi:10.3389/fmars.2025.1674863 · Cited by 2
Jeong, H. (2025) · Asia Pacific Journal of Education · doi:10.1080/02188791.2025.2580988 · Cited by 4
Schepers, I., Bruijn, L. M., Wieling, M. B., & Vols, M. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09495-1 · Cited by 3
A six-phase AI-expert framework for evaluating policy coherence in sustainable tourism
Carić, H., Mandić, A., & Sever, I. (2025) · Information Technology & Tourism · doi:10.1007/s40558-025-00352-0 · Cited by 5
“…content analysis, Delphi-based consensus building, or grounded thematic coding, provide rich contextual insights but are labor-intensive, subjective, and difficult to replicate across large policy corpora (Grimmer and Stewart 2017 ; Lacy et al. 2015 ; Rowe and Wright 1999; Decrop and Masset 2018 ).”
Rizzuni, A. M., Lundgaard, D., Torres‐Cuello, M. A., & Vallentin, S. (2026) · Business Strategy and the Environment · doi:10.1002/bse.70519 · Cited by 3
Can the tone of central bankers’ speeches help shape inflation expectations?: Evidence from Japan
Cho, D., & Rho, S. (2026) · Journal of International Financial Markets Institutions and Money · doi:10.1016/j.intfin.2025.102283 · Cited by 3
Yang, Y. Y.-H., Wang, C., Chou, P.-L., Tien, C.-Y., Wen, J.-C., & Chen, N.-H. (2026) · Computers in Human Behavior · doi:10.1016/j.chb.2026.108904 · Cited by 4
Amosh, H. A. A. (2026) · Corporate Governance An International Review · doi:10.1111/corg.70037 · Cited by 4
11.3. Medical recipe collections
Taavitsainen, I., & Lehto, A. (2019) · John Benjamins Publishing Company eBooks · doi:10.1075/z.221.11.3.taa · Cited by 3
Mishra, R. B., & Jiang, H. (2021) · Sustainability · doi:10.3390/su14010159 · Cited by 3
Lynch, D. C., & Yau, C. W.-K. (2022) · Journal of East Asian Studies · doi:10.1017/jea.2021.37 · Cited by 3
“There is no “gold standard” for obtaining a “best fit” model in STM (Grimmer and Stewart 2013, 19–20).”
Backgroundvan Dijk, F., Gadellaa, J., van Toledo, C., Spruit, M. R., Brinkkemper, S., & Brinkhuis, M. J. S. (2023) · Organizational Cybersecurity Journal Practice Process and People · doi:10.1108/ocj-11-2021-0034 · Cited by 3
Exploring examinees' responses to constructed response items with a supervised topic model
Kim, S., Lu, Z. L., & Cohen, A. S. (2023) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12319 · Cited by 3
Çetintaş, H. B. (2023) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub259.c1179 · Cited by 3
Are Lawyers’ Case Selection Decisions Biased? A Field Experiment on Access to Justice
Frankenreiter, J., & Livermore, M. A. (2023) · The Journal of Legal Studies · doi:10.1086/723282 · Cited by 3
“With the increasing use of high-dimensional data sets in social science research, considerable effort in data transformation and variable selection is often necessary to translate research questions into testable hypotheses (Grimmer and Stewart 2013).”
BackgroundPolarisation and Disinformation Content from Spanish Political Actors on Twitter/X
Said-Hung, E., Merino-Arribas, A., & Martínez-Torres, J. (2023) · Communication Today · doi:10.34135/communicationtoday.2023.vol.14.no.2.8 · Cited by 3
Speaker Attribution in German Parliamentary Debates with QLoRA-adapted Large Language Models
Bornheim, T., Grieger, N., Blaneck, P. G., & Bialonski, S. (2024) · LDV-Forum/Journal for language technology and computational linguistics · doi:10.21248/jlcl.37.2024.244 · Cited by 9
“Language is central to the study of politics, as it forms the basis for political speech and debates (Grimmer and Stewart, 2013).”
BackgroundMining for Meaning : using computational text analysis for social inquiry
Bodell, M. H. (2024) · Institute for Analytical Sociology dissertation series · doi:10.3384/9789180756181 · Cited by 4
Twitter Sentiment Analysis Tweets Using Hugging Face Harnessing NLP for Social Media Insights
Jayalakshmi, V., & Lakshmi, M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-59097-9_28 · Cited by 4
Digital Methods in Economic History: The Case of Computational Text Analysis
Wehrheim, L. (2024) · Springer eBooks · doi:10.1007/978-3-031-35583-7_118 · Cited by 3
The Rise of Sustainability Provisions in International Investment Agreements
Dotzauer, M., Biber‐Freudenberger, L., & Dietz, T. (2024) · Global Environmental Politics · doi:10.1162/glep_a_00759 · Cited by 3
Automated Identification of Business Models
Milei, P., Votintseva, N., & Barajas, Á. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103893 · Cited by 3
“Automated content analysis methods have demonstrated effectiveness on many issues (Grimmer & Stewart, 2013).”
MethodWhere do parties interact? Issue engagement in press releases and tweets
Ivanusch, C. (2024) · European Journal of Political Research · doi:10.1111/1475-6765.12729 · Cited by 3
“While some approaches are particularly useful when topic categories are known beforehand (e.g., supervised learning), others (e.g., unsupervised learning) have their strength in exploration (Grimmer & Stewart, 2013).”
Smart cities lighting the way: Optimizing energy structure and efficiency in the border areas
Zhang, H., Wang, Y., & Liu, H. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.144481 · Cited by 4
Makhortykh, M., de León, E., Christner, C., Sydorova, M., Urman, A., Adam, S., Maier, M., & Gil‐López, T. (2025) · Quality & Quantity · doi:10.1007/s11135-024-02034-1 · Cited by 3
“Preprocessing decreases the amount of noise in textual data by reducing the complexity of textual features (Grimmer and Stewart 2013), which is particularly important for dealing with multi-platform data, where the amount of noise is higher compared with monoplatform data.”
AI in the Public Eye: Decoding Perception of Generative AI Through Natural Language Processing
Jeong, H., & Sung, M. (2025) · Asian Communication Research · doi:10.20879/acr.2025.22.006 · Cited by 3
Biba, B.-J., & O’Shea, B. A. (2025) · Journal of Psychoactive Drugs · doi:10.1080/02791072.2025.2511750 · Cited by 3
Chen, B., Cao, X., & Borah, P. (2025) · Atlantic Journal of Communication · doi:10.1080/15456870.2025.2528785 · Cited by 5
“The choice of a 10% sample is consistent with practices in computational social science and natural language processing research, where validation subsets typically range from 5% to 20% of the total dataset (Grimmer & Stewart, 2013).”
Digital communication strategies of lobbies in the European Union
Lozano, E. S. (2025) · Frontiers in Political Science · doi:10.3389/fpos.2025.1543123 · Cited by 3
Yu, S., Guo, Y., Geng, H., & Hu, X. (2025) · Energy Economics · doi:10.1016/j.eneco.2025.108846 · Cited by 9
Extrusive effects of digital government: Reframing the citizen–state relationship
Mota, E. C., & Klein, P. G. (2025) · Government Information Quarterly · doi:10.1016/j.giq.2025.102064 · Cited by 8
Topics as Outcomes: Modeling the Influence of Intergovernmental Grants on Policy Diffusion
Brodnax, N., & James, S. W. (2025) · Policy Studies Journal · doi:10.1111/psj.70074 · Cited by 4
“Automated text analysis is increasingly used to process, categorize, and analyze political corpora (Grimmer and Stewart 2013).”
Computerlinguistische Textanalyse und Debatten im Parlament
Heiberger, R. H., & Koß, C. (2017) · Springer eBooks · doi:10.1007/978-3-658-19945-6_15 · Cited by 3
Contrastive Training for Models of Information Cascades
Xu, S., & Smith, D. A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11270 · Cited by 4
“If you read and listen to all these statements, you might notice common tropes and turns of phrase that groups of politicians used to describe some issue (Grimmer and Stewart 2013).”
BackgroundJoint Sentence-Document Model for Manifesto Text Analysis
Subramanian, S., Cohn, T., Baldwin, T., & Brooke, J. (2017) · Cited by 3
CoUSBi : A Structured and Visualized Legal Corpus of US State Bills
Kalouli, A.-L., Vrana, L., Fabella, V. M., et al. (2018) · Language Resources and Evaluation · Cited by 3
Monitoring rare categories in sentiment and opinion analysis: a Milan mega event on Twitter platform
Calissano, A., Vantini, S., & Arena, M. (2019) · Statistical Methods & Applications · doi:10.1007/s10260-019-00504-7 · Cited by 3
“Classification methods, instead, organize texts into a set of categories, supposing that the meaning (and sentiment) of the text is given by considering the word combination (Grimmer and Stewart 2013).”
MethodInfluentialThe Federal Justice act in combating corruption in Southern Brazil
Madeira, L. M., & Geliski, L. (2019) · Revista de Administração Pública · doi:10.1590/0034-761220180237x · Cited by 3
From Reproducible to Productive
Goldstone, A. (2017) · Journal of Cultural Analytics · doi:10.22148/001c.11821 · Cited by 3
Beyond Citations: Corpus-based Methods for Detecting the Impact of Research Outcomes on Society
Rezapour, R., Bopp, J., Fiedler, N., Steffen, D., Witt, A., & Diesner, J. (2020) · Publication Server of the Institute for German Language (Institute for German Language) · Cited by 3
Phillips, J. B., & Woodman, V. K. G. (2020) · Australian Journal of Political Science · doi:10.1080/10361146.2020.1774506 · Cited by 3
Overall Trends of Protest in the Great Recession
Kriesi, H. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108891660.005 · Cited by 3
Topic Mining and Effectiveness Evaluation of China’s Coal-Related Policy Based on LDA Model
Yue, F., Guo, K., Yue, M., et al. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-8599-9_66 · Cited by 3
Pareschi, L. (2021) · puntOorg International Journal · doi:10.19245/25.05.pij.6.2.5 · Cited by 3
Zhang, Z. (2021) · Journal of Multilingual and Multicultural Development · doi:10.1080/01434632.2021.1938087 · Cited by 3
“…its features, topic modelling can serve as a bridge in the social sciences, for it sits at the interfaces between case studies and big data, unstructured and structured analysis, and induction and deduction (DiMaggio, Nag, and Blei 2013; Grimmer and Stewart 2013; Mützel 2015; Hannigan et al. 2019).”
BackgroundZhang, Z. (2021) · HERMES - Journal of Language and Communication in Business · doi:10.7146/hjlcb.vi61.127928 · Cited by 3
Measuring political legitimacy with Twitter: Insights from India’s Aadhaar program
Mahrenbach, L. C., & Pfeffer, J. (2021) · New Media & Society · doi:10.1177/14614448211033493 · Cited by 5
Extracting video games rating labels from transcript files
Maisto, A., Martorelli, G., Paone, A., et al. (2021) · Internet of Things · doi:10.1016/j.iot.2021.100439 · Cited by 3
Heeb, R., Taff, S. D., & Bakhshi, P. (2021) · Annals of International Occupational Therapy · doi:10.3928/24761222-20210921-03 · Cited by 3
Browne, I., Bernau, J. A., Tatum, K., & Jieyu, J. (2021) · Sociological Quarterly · doi:10.1080/00380253.2021.1947757 · Cited by 4
“While there is no “right answer” to the number of topics used to analyze a corpus (Grimmer and Stewart 2013), Roberts, Stewart, and Tingley (2019) identify a suite of approximate “goodness of fit” measures for a range of possible topic values.”
MethodIS JAPAN BACK? MEASURING NATIONALISM AND MILITARY ASSERTIVENESS IN ASIA'S OTHER GREAT POWER
Lind, J. M., & Ueki, C. K. (2021) · Journal of East Asian Studies · doi:10.1017/jea.2021.20 · Cited by 4
“On problems with automatic coding methods (exacerbated when used on texts from different languages and cultures) see Krebs 2015, 199–200; Grimmer and Stewart 2013.”
MethodParteiendifferenz in der Waldpolitik. Eine Analyse der deutschen Bundesländer
Hartung, U., Jankowski, M., & Müller, J. M. (2021) · Zeitschrift für Vergleichende Politikwissenschaft · doi:10.1007/s12286-021-00495-4 · Cited by 3
Pension Policy and Governmentality in China: Manufacturing Public Compliance
Wang, Y. (2022) · LSE Press eBooks · doi:10.31389/lsepress.ppc · Cited by 3
Computergestützte Textanalysen
Proksch, S. (2020) · Springer eBooks · doi:10.1007/978-3-658-16936-7_38 · Cited by 3
Jin, B., & Kirsch, D. A. (2022) · Industry and Innovation · doi:10.1080/13662716.2022.2114318 · Cited by 3
“Some scholars point out that it is crucial to validate the output of this method (e.g. Grimmer and Brandon 2013).”
MethodAnalysis of Social Media Utilization based on Big Data-Focusing on the Chinese Government Weibo
Li, X., Guo, X., Kim, S. K., & Lee, H. (2022) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2022.08.006 · Cited by 3
Extracting and Quantifying Actionable Knowledge Using Twitter Data
Ramesh, B., & Weber, C. M. (2022) · Portland International Conference on Management of Engineering and Technology (PICMET) · doi:10.23919/picmet53225.2022.9882680 · Cited by 3
“There has been increasing importance of computational text analysis in communication research [43], [44] in the last decade.”
BackgroundZuo, C. (2022) · Studies in Comparative International Development · doi:10.1007/s12116-022-09373-w · Cited by 4
“Following Grimmer and Stewart (2013), we randomly pick 10% of the text and hand code the geographic focus.”
MethodOliveira, I., Nogueira, F., & Marques, I. C. P. (2022) · New Trends in Qualitative Research · doi:10.36367/ntqr.11.2022.e533 · Cited by 3
Content Analysis in the Research Field of Video Games
Wulf, T., Possler, D., & Breuer, J. (2022) · Springer eBooks · doi:10.1007/978-3-658-36179-2_25 · Cited by 3
Do Journals Have Preferences? Insights from The Journal of Higher Education
Havey, N. F., & Chang, M. J. (2022) · Innovative Higher Education · doi:10.1007/s10755-022-09634-5 · Cited by 3
“…a topic number K by minimizing topic redundancy, maximizing semantic coherence of the output topics, and emphasizing topic uniqueness (i.e., merging topics like State Funding and State Allocations) to ensure a broad range of topics that had minimal overlap (Grimmer, 2010; Grimmer & Stewart, 2013).”
BackgroundFranco‐Guillén, N., Laile, S. D., & Parkinson, J. (2022) · doi:10.1093/oso/9780192848925.003.0016 · Cited by 3
Schneider, G. (2022) · Transactions of the Philological Society · doi:10.1111/1467-968x.12252 · Cited by 3
Framing School Integration: A Computational Analysis of Newspaper Discourse, 1990–2020
Diehl, D. K. (2022) · Social Problems · doi:10.1093/socpro/spac052 · Cited by 3
Tweeting for Latinos? Legislative Communication in the 115th U.S. House of Representatives
Pleites‐Hernandez, G. D. (2023) · Congress & the Presidency · doi:10.1080/07343469.2022.2129522 · Cited by 3
“Instead of coding each one of those tweets by hand, I chose to rely on a method that is increasingly used by scholars in the field: supervised machine learning (see Grimmer and Stewart 2013; Grimmer 2016; Lassen and Bode 2017).”
MethodSit Still, Talk Pretty: Partisan Differences Among Women Candidates’ Campaign Appeals
Russell, A., Macdonald, M., & Hua, W. (2023) · Journal of Women Politics & Policy · doi:10.1080/1554477x.2023.2194232 · Cited by 3
“In contrast to hand-coding, automated textual analysis allows us to study congressional communications more comprehensively and efficiently (see: Grimmer and Stewart 2013).”
BackgroundMethodMachine learning meets the Journal of Public Budgeting and Finance: Topics and trends over 40 years
Chen, C., Xiao, S., & Zhao, B. (2023) · Public Budgeting & Finance · doi:10.1111/pbaf.12348 · Cited by 3
Lattimer, T. A., Tenzek, K. E., & Ophir, Y. (2023) · Health Communication · doi:10.1080/10410236.2023.2250939 · Cited by 3
“Finally, out methodology also included extensive human involvement to analyze themes and provide both face and semantic validity checks (Grimmer & Stewart, 2013; Quinn et al., 2010).”
Framing the Pandemic on Persian Twitter: Gauging Networked Frames by Topic Modeling
Kermani, H. (2023) · American Behavioral Scientist · doi:10.1177/00027642231207078 · Cited by 3
“This study reinforces previous research that suggests computational methods are more effective with straightforward and general concepts (De Grove et al., 2020; Grimmer & Stewart, 2013).”
Chang, W.-C. (2023) · Computer Science and Information Systems · doi:10.2298/csis220830069c · Cited by 3
Natural Language Decompositions of Implicit Content Enable Better Text Representations
Hoyle, A., Sarkar, R., Goel, P., & Resnik, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.815 · Cited by 10
“…human interpretation of text by aggregating lexical data; for example, dictionaries map words to constructs (e.g., Pennebaker et al., 2001), and topic models discover interpretable categories in a form of automated qualitative content analysis (Grimmer and Stewart, 2013; Hoyle et al., 2022).”
MethodYeung, R. C., & Fernandes, M. A. (2023) · npj Mental Health Research · doi:10.1038/s44184-023-00042-x · Cited by 5
“We then selected a fi nal number of topics using a two-stage approach 41,50 .”
BackgroundMethodInfluentialChen, W., Zhang, H., & Hu, Y. (2024) · Cities · doi:10.1016/j.cities.2024.104900 · Cited by 3
Finding the needle in the haystack: archival research in European political science
Klein, A. (2024) · European Political Science · doi:10.1057/s41304-024-00488-3 · Cited by 5
“The mere scanning of documents takes on a different meaning if researchers employ a text-as-data approach for which archival documents are suitable (Grimmer and Stewart 2013; Wilkerson and Casas 2017).”
Siefken, L. (2024) · German Politics · doi:10.1080/09644008.2024.2355211 · Cited by 4
“…data sources, such as speeches, news reports, or campaign material to measuring different concepts such as ideological positions on specific topics or issue salience in politics (Grimmer and Stewart 2013; Laver, Benoit, and Garry 2003; Lucas et al. 2015; Volkens et al. 2021; Young and Soroka 2012).”
Acampa, S., & Nunziata, F. (2024) · Social Sciences · doi:10.3390/socsci13060292 · Cited by 5
Overgaard, C. S. B., Lukito, J., & Soorholtz, K. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31380 · Cited by 9
Wright, K. (2024) · British Politics · doi:10.1057/s41293-024-00257-2 · Cited by 3
Mao, J., Wang, D., Chen, F. W., et al. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2024.123690 · Cited by 3
Uncovering Xi Jinping's Policy Agenda: Text As Data Approach
Lim, J., Ito, A., & Zhang, H. (2024) · The Developing Economies · doi:10.1111/deve.12418 · Cited by 3
Cardozo‐Gaibisso, L., Hodges, G. W., Mardones-Segovia, C., & Cohen, A. S. (2024) · Education Sciences · doi:10.3390/educsci14101068 · Cited by 3
Makhortykh, M., de León, E., Urman, A., Gil‐López, T., Christner, C., Sydorova, M., Adam, S., & Maier, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0312865 · Cited by 3
Yang, Y. (2025) · Oxford University Press eBooks · doi:10.1093/9780197820339.001.0001 · Cited by 4
Applications of latent variable models in modeling influence and decision making
Blei, D. M., & Gerrish, S. (2013) · Cited by 3
Mitchell, A., Lemon, M., & Fletcher, G. (2018) · The Central European Review of Economics and Management · doi:10.29015/cerem.596 · Cited by 3
A case study in text mining: Textual analysis of the Territorial Papers
Ledolter, J., & VanderVelde, L. S. (2019) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqz007 · Cited by 3
Ovádek, M., Lampach, N., & Dyèvre, A. (2020) · European Union Politics · doi:10.1177/1465116520902530 · Cited by 2
“For that reason, we rely on automated content analysis techniques to estimate the latent topical structure of our corpus (Catalinac, 2016; Dybowski and Ad€ammer, 2018; Grimmer, 2010; Grimmer and Stewart, 2013; Kim, 2018; Quinn et al., 2010; Rothschild et al., 2019).”
InfluentialGroup Conflicts in Light of the Cratism Theory (Psycholinguistic Analysis)
Citlak, A. (2020) · Roczniki Psychologiczne · doi:10.18290/rpsych20232-1 · Cited by 4
Integrating human and machine coding to measure political issues in ethnic newspaper articles
Kim, J. Y. (2021) · Journal of Computational Social Science · doi:10.1007/s42001-020-00097-2 · Cited by 3
“Although computational text analysis has gained popularity among social scientists [9, 42, 110], empirical research that closely investigates humancoding decisions’ impact on automated text classification remains lacking [28, 30, 84–86].”
BackgroundUsing Text Mining For Research Trends in Empirical Software Engineering
Tokdemir, G. (2021) · Journal of Polytechnic · doi:10.2339/politeknik.831391 · Cited by 3
Metadata
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
