The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
161 Own interpretation · 38 No basis given · 28 Sources not found · 26 No full text of the sources · 21 Own result · 11 Supported · 8 Not checked yet · 1 Partly supported · 1 Own method
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Supportedcites Hand 2006 source
Section 3, “Design Sample Selection,” says the classical supervised-classification paradigm assumes the design set is randomly drawn from the same distribution as future points to be classified. That supports the citing sentence’s application of the assumption to documents; the supplied copy does not provide a page number. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract says that determining in advance which clustering objective function will produce a useful or insightful partition for a given dataset is unknown and difficult, if not logically impossible. It also says there is no method for choosing beforehand which unsupervised approach will yield the most useful clustering for a particular application. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
In Model Output (page unavailable), the paper calls its speech-topic assignments “parameters of direct interest” and uses them to measure attention to political topics over time and by speaker group. Model Specification and Sensitivity Analysis (page unavailable) says the authors selected topic categories to correspond to areas of governmental competence. Together, these support the paraphrase. (LLM, conf=0.87)
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
The model description explains that speech topics are inferred and their probabilities vary by day. The Abstract reports a daily series of attention to topics in U.S. Senate speeches. These passages support the quoted claim; they do not establish the separate 'first topic model' characterization in the citing sentence. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology (§2.4) says the authors project a metric space of clusterings into two Euclidean dimensions and that nearby points represent similar clusterings. Methodology (§2.3) defines similarity by how documents are grouped across clusterings, matching the citing sentence’s paraphrase. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Categorizing Texts: Methods, Assumptions, and Costs” (page not available), the paper says its topic model infers categories rather than requiring them to be fixed beforehand, and shifts effort to postanalysis. “Topic Interpretation and Intratopic Semantic Validity” (page not available) describes examining the resulting topics and documents to assess their validity. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“empirically these sentences are rare.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the restrained category, the recall is 0.75.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 158
- It may state two separate claims as one. 34
Every source the paper cites for this claim was checked and supports it.
In the paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Supportedcites Berinsky, Huber, and Lenz 2012 source
The Introduction says MTurk can facilitate low-cost experiments, and Section 2 describes how researchers post tasks for workers and recruit survey participants. The Abstract also describes MTurk experiments as easy to field. Together, these passages support the claim that the platform can make data acquisition efficient. (LLM, conf=0.91)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Supportedcites Monroe, Colaresi, and Quinn (2008) source
Section 3 describes methods for selecting and evaluating words that distinguish Democratic from Republican Senate speeches. Section 3.1 makes clear that the categories are already identified: the speakers’ partisanship is observed, not something the methods need to infer. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Grimmer and King 2011 source
Section 3.1 (page unavailable) defines cluster quality from human judgments as within-cluster similarity minus between-cluster similarity, and Section 3 reports using that measure to compare clusterings. The measure is comparative rather than an absolute test of semantic validity, consistent with the paper’s caution that there is no reliable gold-standard clustering for this purpose. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract (p. 2643) says the authors develop a space containing existing clustering methods and millions of additional solutions based on combinations of existing clusterings, then enable users to explore it quickly. Section 2, Methodology, describes how users navigate that space to select useful clusterings. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology section describes CAC as organizing a large number of possible clusterings in a visual space so researchers can efficiently select useful ones. The Abstract likewise says users can explore a space containing existing methods and millions of additional solutions and quickly identify useful clusterings. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Exogenous Events and Predictive Validity,” the paper defines predictive validity as correspondence between a measure and external events, then links spikes in topic attention to events including 9/11 and debates over military action. The Discussion says examining relationships with such events helps assess the validity of its attention measures. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The topics in Table 4 exemplify the utility of unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language is the medium for politics and political conflict.”
- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Politics and political conflict often occur in the written and spoken word.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential when applying automated content methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it substantially reduces the costs of analyzing large collections of text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“to understand what politics is about we need to know what political actors are saying and writing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“require extensive and problem-specific validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they are no substitute for careful thought and close reading”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of any one classifier can vary substantially across contexts”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“And hiring coders to manually read all documents is still very expensive.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One way to improve upon dictionaries are supervised methods”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“automated content methods are incorrect models of language.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding the separating words is also relatively easy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tone is just one type of analysis a dictionary method can perform.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is often difficult, and sometimes impossible, to validate the output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe described above is one way to represent text as data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather than replace humans, computers amplify human abilities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is also substantial variation within families of models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validations are necessary to establish its meaning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- The authors state this as an assumption: Ideology dominates the language used in the text.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The data generation process for any text (including this one) is a mystery-even to linguists.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of applied research uses stemming.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“sometimes the problem calls for a completely different approach.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stemmer is a much cruder algorithm, but considerably faster.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance does not seem to matter for most applications”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“automated scraping methods make acquiring these data easier (Jackman 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These programs simply provide the researcher with output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying FAC algorithms to texts may have mixed results”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
What the sources say
- Not checked — no full text of the sourcecites Spirling (2012) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Eggers and Hainmueller 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated methods also rely on having a sufficient number of words to use reliably.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Porter 1980 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Not matchedcites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these require the close reading of texts and thoughtful analysis by the researcher.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assigning texts to categories is the most common use of content analysis methods in political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“not all of language's complexity is necessary to effectively analyze texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
What the sources say
- Not matchedcites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.3
What the sources say
- Not checked — no full text of the sourcecites van der Vaart, Dudoit, and van der Laan 2006 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Not matchedcites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
- Not checked — no full text of the sourcecites Hastie, Tibshirani, and Friedman 2001 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Dictionary methods require scholars to identify words that separate classes beforehand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number necessary will depend upon the specific application of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
- Not checked — no full text of the sourcecites Hillard, Purpura, and Wilkerson 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAC methods are useful, but are necessarily model dependent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1 FAC
What the sources say
- Not matchedcites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- The authors state this as an assumption: Given a document's category, the words are generated independently.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Supervised and dictionary methods assume a well-defined set of categories.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Yiannakis 1982 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the K-means' objective function is multimodal and non-continuous.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not checked — no full text of the sourcecites Blei 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and therefore several random restarts are often necessary to find the global optimum.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § 6.1.2 Mixed membership models · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The goal is revelation of substantively interesting information.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To estimate a topic, the models use the co-occurrence of words across documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Situating the models in a data-generating process makes including additional structure straightforward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preprocessed texts represent a substantial simplification of the documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CAC methods provide a way to explore these different assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, CAC provides an accelerated way to explore new collections of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many other distance metrics are possible, and each will lead to different clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But CAC does place substantial burden on users to identify clusterings that are useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.2 Unsupervised Methods for Scaling
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Lowe 2008 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But nearly all political actors speak.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We caution that this validation does not obviate the need for unsupervised methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that any one method's performance will be context specific.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimating locations using existing data is often difficult and sometimes impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not matchedcites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Experimental — accuracy not yet measuredIn the citing paper
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
In the citing paper
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 25,941
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
(1967) · Cited by 23,719
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Items 1,401–1,500 of 3,308
Non, N., & Aziz, N. A. (2023) · Journal of financial reporting & accounting · doi:10.1108/jfra-10-2022-0364 · Cited by 7
Kermani, H., Makou, A. B., Tafreshi, A., et al. (2023) · International Journal of Social Research Methodology · doi:10.1080/13645579.2023.2186566 · Cited by 7
“Another advantage of LDA is its mixed membership approach (Grimmer & Stewart, 2013, p. 18).”
BackgroundResultInfluentialHase, V., Mahl, D., & Schäfer, M. S. (2023) · Online Media and Global Communication · doi:10.1515/omgc-2023-0003 · Cited by 7
“For an overview of the method, we refer readers to Benoit (2020), Grimmer and Stewart (2013), and Gün-ther and Quandt (2016) as well as to Williams et al. (2020) for the analysis of visual content.”
BackgroundMethodZhao, H., Chen, W., Zhang, C., et al. (2023) · Natural hazards and earth system sciences · doi:10.5194/nhess-23-3031-2023 · Cited by 7
“tions in a variety of natural and social sciences (Quackenbush, 2002; Tetlock, 2007; Grimmer and Stewart, 2013; Law et al., 2014).”
BackgroundExploring the nexus of organisational culture and sustainability for green innovation
Demastus, J., Ohsowski, B. M., & Landrum, N. E. (2024) · Industry and Innovation · doi:10.1080/13662716.2024.2390991 · Cited by 7
“Fully Automated Clustering was also applied, meaning no algorithms were used to induce data beyond word count (Justin and Stewart 2013; Lucas et al. 2015).”
Frost durability of cementitious materials: What’s next?
Rajczakowska, M., Nováková, I., Adediran, A. A., et al. (2024) · Case Studies in Construction Materials · doi:10.1016/j.cscm.2024.e04014 · Cited by 7
Sun, W., Hu, S., & Shen, L. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.145021 · Cited by 7
Zhang, Y., Huang, B., Yuan, W., Jiang, Z., Peng, L., Chen, S., & Tan‐Soo, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104090 · Cited by 7
L’Économie Sociale et Solidaire et ses parties prenantes : une étude du discours
Mariaux, S., & Reynaud, E. (2018) · Finance Contrôle Stratégie · doi:10.4000/fcs.2551 · Cited by 7
Houborg, E., & Enghoff, O. (2018) · Tidsskrift for Forskning i Sygdom og Samfund · doi:10.7146/tfss.v15i28.107265 · Cited by 7
European Rural Development Policy Approaching Health Issues: An Exploration of Programming Schemes
Samoggia, A., Bertazzoli, A., & Ruggeri, A. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16162973 · Cited by 7
Rally around the EU flag: Irish party positions on the EU in the wake of Brexit
Costello, R. (2020) · Journal of Contemporary European Studies · doi:10.1080/14782804.2020.1825352 · Cited by 7
“This is consistent with the recommendations in the literature on measuring political positions from parliamentary debates, which stresses the importance of isolating procedural contributions from substantive debates (e.g. Lauderdale and Herzog 2016, p376; Grimmer and Stewart 2013, 294).”
ResultIzumi, M., & Medeiros, D. B. (2021) · Latin American Politics and Society · doi:10.1017/lap.2020.36 · Cited by 7
“The use of text as data has grown continuously in political science, which constitutes an important contribution, because political conflict often occurs in written and spoken words (Monroe and Schrodt 2009; Grimmer and Stewart 2013).”
BackgroundHarnessing Technologies in Focus Group Research
Wallace, R., Goodyear‐Grant, E., & Bittner, A. (2021) · Canadian Journal of Political Science · doi:10.1017/s0008423921000226 · Cited by 7
“8 Although automated analyses are very efficient and effective, they can produce errors and hence require extensive validation (Lowe and Benoit, 2013; Grimmer and Stewart, 2013).”
New evidence and new methods for analyzing the Iranian revolution as an intelligence failure
Connelly, M., Hicks, R., Jervis, R., & Spirling, A. (2021) · Intelligence & National Security · doi:10.1080/02684527.2021.1946959 · Cited by 10
A machine learning approach to the study of German strategic culture
Tappe, J., & Doeser, F. (2021) · Contemporary Security Policy · doi:10.1080/13523260.2021.1992150 · Cited by 7
PTPARL-D: an annotated corpus of forty-four years of Portuguese parliamentary debates
Almeida, P., Marques-Pita, M., & Gonçalves-Sá, J. (2021) · Corpora · doi:10.3366/cor.2021.0226 · Cited by 7
“Several tools have been developed to automatically analyze complex and large political texts, which increasingly include not only manifestos but also parliamentary addresses, debates and commentary [6, 13, 12], and these techniques have proven themselves, mainly focusing on the much simpler two-party US system.”
MethodFaraway, so close: a spatial account of the Conte I government formation in Italy, 2018
Giannetti, D., Pedrazzani, A., & Pinto, L. (2021) · Italian Political Science Review/Rivista Italiana di Scienza Politica · doi:10.1017/ipo.2021.11 · Cited by 7
“More recently, a new area of research has started to apply automated and statistical approaches to scale positions from political ‘text as data’ (Grimmer and Stewart, 2013).”
BackgroundKovář, J. (2022) · Europe Asia Studies · doi:10.1080/09668136.2022.2061426 · Cited by 7
“…of accuracy with regard to an understanding of textual meanings and the flexibility to discover new frames inductively during the coding process (Grimmer & Stewart 2013).4 While training the coders, I revised the initial coding scheme upon the completion of the coding of the first 200 items to…”
MethodTrends and Opportunities for Bridging Prevention Science and US Federal Policy
Scott, T., Prendergast, S., Demeusy, E. M., McGuire, K., & Crowley, D. M. (2022) · Prevention Science · doi:10.1007/s11121-022-01403-2 · Cited by 8
“Since bills are sometimes very lengthy, only sections related to research use and juvenile justice or delinquency were coded and analyzed (Grimmer & Stewart, 2013).”
BackgroundCollecting migrants' Facebook posts: Accounting for ethical measures in a text-as-data approach
Gertz, H. D. (2023) · Frontiers in Sociology · doi:10.3389/fsoc.2022.932908 · Cited by 7
“The applied “text as data” approach (Grimmer and Stewart, 2013) follows a standardized and, theoretically, reproducible methodology while complying withmeasures for data protection and having no financial profit.”
MethodSentiment Analysis of COVID-19 Tweets Using TextBlob and Machine Learning Classifiers
Kathiravan, P., Saranya, R., & Sekar, S. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-19-6634-7_8 · Cited by 7
A tale of heroes and villains: Russia’s strategic narratives on Twitter during the COVID-19 pandemic
Moral, P. (2023) · Journal of Information Technology & Politics · doi:10.1080/19331681.2023.2182862 · Cited by 13
“Lastly, the present study seeks to ensure the validity of Bertopic and its posterior analysis through “external validation,” verifying that attention to topics responds in predictable ways to events that should affect their prevalence (Di Maggio et al., 2013; Grimmer & Stewart, 2011).”
MethodAmores, J. J., Blanco‐Herrero, D., & Calderón, C. A. (2023) · Journalism and Media · doi:10.3390/journalmedia4020030 · Cited by 7
Riveter: Measuring Power and Social Dynamics Between Entities
Antoniak, M., Field, A., Mun, J., Walsh, M., Klein, L. F., & Sap, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.36 · Cited by 14
“They are interpretable and intuitive (Grimmer and Stewart, 2013), especially when compared with black-box classification models, and continue to be a go-to resource (e.g., LIWC; Pennebaker et al., 2015).”
MethodMeasuring Political Narratives in African News Media: A Word Embeddings Approach
Kitagawa, R., & Shen-Bayh, F. F. (2023) · The Journal of Politics · doi:10.1086/727593 · Cited by 7
“However, such measures invoke the bag-of-words assumption: word order is ignored, which can result in lost contextualization when summarizing texts (Grimmer and Stewart 2013).”
BackgroundVice versa: The decoupling of content and topic heterogeneity in collusion research
Schmal, W. B. (2023) · Journal of Economic Surveys · doi:10.1111/joes.12600 · Cited by 6
“There is no ‘one size fits all’ approach that provides a unique set of metrics to elicit the optimal K∗ (see, e.g., Grimmer & Stewart, 2013; Wehrheim, 2019).”
BackgroundFormative Ideas of the National University: A Thematic Analysis of Texts from the Long 19th Century
Bengson, N. (2024) · Higher Education Policy · doi:10.1057/s41307-023-00342-z · Cited by 7
“Future reproductions of this study could benefit from future computational tools for large-scale textual analysis (see Grimmer and Stewart 2013; Baden et al. 2022) and, as noted, further inclusion of texts in other world languages.”
Review of ambiguity problem in text summarization using hybrid ACA and SLR
Sutriawan, S., Rustad, S., Shidik, G. F., Pujiono, P., & Muljono, M. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200360 · Cited by 15
The Role of Language in the Automatic Coding of Political Texts
Ruedin, D. (2013) · Swiss Political Science Review · doi:10.1111/spsr.12050 · Cited by 7
“There has been significant progress on automatic coding of political texts in recent years, and such approaches are becoming more commonplace (e.g. Laver and Garry, 2000; Laver et al., 2003; Grimmer, 2010; Grimmer and King, 2011; Grimmer and Stewart, 2013).”
BackgroundUnderstanding the network fundamentals of news sources associated with a specific topic
Park, D., Kim, G.-N., & On, B.-W. (2016) · Information Sciences · doi:10.1016/j.ins.2016.08.015 · Cited by 7
Representations of electronic cigarettes in Chinese media
Sun, S., Veltri, G., & Wang, F. (2018) · BMC Public Health · doi:10.1186/s12889-018-5644-x · Cited by 7
“Such an automatic text analysis involves three steps including constructing training data, classifying articles, and validating results [23].”
MethodVoter discounting of party campaign manifestos
Fernández-Vázquez, P. (2018) · Party Politics · doi:10.1177/1354068818787352 · Cited by 7
Findings from the hackathon on understanding euroscepticism through the lens of textual data
Nanni, F., Glavaš, G., Ponzetto, S. P., Tonelli, S., Conti, N., Aker, A., Aprosio, A. P., Bleier, A., Carlotti, B., Gessler, T., et al. (2018) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 8
Target Based Speech Act Classification in Political Campaign Text
Subramanian, S., Cohn, T., & Baldwin, T. (2019) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s19-1030 · Cited by 7
“The recent adoption of NLP methods has led to significant advances in the field of computational social science (Lazer et al., 2009), including political science (Grimmer and Stewart, 2013).”
BackgroundTime-dependent Poisson reduced rank models for political text data analysis
Jentsch, C., Lee, E. R., & Mammen, E. (2019) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2019.106813 · Cited by 7
Liu, J., O’Donnell, M. B., & Falk, E. B. (2020) · Health Communication · doi:10.1080/10410236.2020.1712521 · Cited by 7
“For example, we performed quantitative linguistic analysis with dictionarybased approach (Grimmer & Stewart, 2013).”
MethodMulti-Label Classification of Historical Documents by Using Hierarchical Attention Networks
Kim, D.-K., Lee, B., Kim, D., & Jeong, H. (2020) · Journal of the Korean Physical Society · doi:10.3938/jkps.76.368 · Cited by 7
Ranking professional forecasters by the predictive power of their narratives
Rybinski, K. I. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.04.003 · Cited by 7
Ishima, H. (2020) · Electoral Studies · doi:10.1016/j.electstud.2020.102192 · Cited by 7
Cordell, R., Gleditsch, K. S., Kern, F. G., & Saavedra-Lux, L. (2020) · Journal of Peace Research · doi:10.1177/0022343320959122 · Cited by 8
“Computerized approaches can be used to estimate actors’ ideal points from political texts or organize and classify documents into pre-defined or new categories (Grimmer & Stewart, 2013).”
BackgroundDeveloping a Curated Topic Model for COVID-19 Medical Research Literature
Resnik, P., Goodman, K. E., & Moran, M. (2020) · doi:10.18653/v1/2020.nlpcovid19-2.30 · Cited by 7
JURI SAYS: An Automatic Judgement Prediction System for the European Court of Human Rights
Medvedeva, M., Xu, X., Wieling, M. B., & Vols, M. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200883 · Cited by 10
Strategic Partisans: Electoral Motivations and Partisanship in Local Government Communication
Benedictis-Kessner, J. D. (2021) · Journal of Political Institutions and Political Economy · doi:10.1561/113.00000036 · Cited by 7
Johnson, T., & Lerner, J. Y. (2021) · Review of International Political Economy · doi:10.1080/09692290.2021.1974523 · Cited by 6
“This matrix enables comparisons of documents (Grimmer & Stewart, 2013).”
BackgroundRetreating from Redistribution? Trends in Democratic Party Fidelity to Economic Equality, 1984–2020
Malpas, A., & Hilton, A. (2021) · The Forum · doi:10.1515/for-2021-0012 · Cited by 7
Extracting semantic relations using syntax
Welbers, K., van Atteveldt, W., & Kleinnijenhuis, J. (2021) · Computational Communication Research · doi:10.5117/ccr2021.2.003.welb · Cited by 7
Guber, D. L. (2021) · Public Opinion Quarterly · doi:10.1093/poq/nfab053 · Cited by 7
On the Use of Crowdsourcing Labor Markets in Research
Marinova, D. M. (2016) · Perspectives on Politics · doi:10.1017/s1537592716000104 · Cited by 7
“17 Grimmer and Stewart 2013.”
Huang, R. (2019) · Chinese Journal of Sociology · doi:10.1177/2057150x18820500 · Cited by 7
English Character Recognition Using Robust Back Propagation Neural Network
Zanwar, S. R., Narote, A. S., & Narote, S. P. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-9187-3_20 · Cited by 7
Schoots, J., Rohr, B., Prajda, K., & Padgett, J. F. (2020) · Poetics · doi:10.1016/j.poetic.2019.101386 · Cited by 7
Nicholson, S., & Bennett, T. (2016) · The International Information & Library Review · doi:10.1080/10572317.2016.1176448 · Cited by 7
“In a similar vein, Grimmer and Stewart (2013) provide a guide for political scientists to automated text analysis for large collections of digital texts.”
BackgroundHuman Rights Texts: Converting Human Rights Primary Source Documents into DataPreprint
Fariss, C. J., Jones, Z. M., Linder, F., et al. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2502980 · Cited by 7
Scaling Up Content Analysis: Crowd-Coding Text UnitsPreprint
Adams, N. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2617222 · Cited by 7
Machine learning and behavioral economics for personalized choice architecturePreprint
Hrnjić, E., & Tomczak, N. (2019) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1907.02100 · Cited by 17
“[34] Justin Grimmer and Brandon M Stewart, “Text as data:”
BackgroundChatsiou, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10267 · Cited by 11
“[12] Justin Grimmer and Brandon M.”
A Statistical Test for Legal Interpretation: Theory and ApplicationsPreprint
Nyarko, J., & Sanga, S. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3737292 · Cited by 7
Demirel, S., & Gündüz, U. (2022) · Intermedia International e-journal · doi:10.56133/intermedia.1163032 · Cited by 6
Malaterre, C., & Lareau, F. (2024) · Data & Policy · doi:10.1017/dap.2023.48 · Cited by 7
What’s in a name? The effect of named entities on topic modelling interpretability
Tolochko, P., Balluff, P., Bernhard-Harrer, J., Galyga, S., Lebernegg, N. S., & Boomgaarden, H. G. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2024.2302120 · Cited by 7
“Topic quality and validation remain a difficult and an ongoing question in related research (e.g., Doogan & Buntine, 2021; Grimmer et al., 2022; Grimmer & Stewart, 2013; Lau et al., 2014, etc.).”
Krauss, S., Praprotnik, K., & Groß, M. (2025) · Party Politics · doi:10.1177/13540688241306732 · Cited by 6
“To receive a party ’ s share of issue attention in the regional manifestos, we used a dictionary approach (see, e.g., Grimmer and Stewart, 2013: 274 – 275).”
Czudaj, R., & Nguyen, B. N. (2025) · Macroeconomic Dynamics · doi:10.1017/s1365100525000239 · Cited by 6
“The dictionary would search for the exact word or synonyms in the window of the target words, assign a value for each word that appeared, and categorize them (Grimmer and Stewart, 2013).”
Finet, A., Kristoforidis, K., & Laznicka, J. (2025) · F1000Research · doi:10.12688/f1000research.164796.1 · Cited by 6
Media choice and audience perceptions: Evidence from visual framing of immigration in news stories
Gasparyan, O., & Сироткина, Е. (2025) · PLoS ONE · doi:10.1371/journal.pone.0331219 · Cited by 6
“The study of framing, though, has historically prioritized textual analysis, partly due to the methodological ease of dissecting written content with automated tools like topic modeling [33] and sentiment analysis [34].”
Almelhem, A., Iyigun, M. F., Kennedy, A., et al. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjaf054 · Cited by 6
Zambrano, J. P. U., Torres, B., De‐Pablos‐Heredero, C., Herrera-Feijoo, R. J., & García, A. R. (2025) · Land · doi:10.3390/land15010013 · Cited by 6
“Semantic normalization procedures were implemented to ensure lexical consistency across the dataset, encompassing the unification of morphological variants, the standardization of singular and plural forms, and the consolidation of synonymous expressions [42,43].”
Perfect scapegoats? Blaming and defending the International Monetary Fund
Heinkelmann‐Wild, T., Hunter, T., & Shim, S. (2025) · The Review of International Organizations · doi:10.1007/s11558-025-09607-1 · Cited by 6
“We opted for manual coding rather than automated methods because it remains the “gold standard” (Grimmer & Stewart, 2013), and if its costs are not prohibitive, it should be the default when performing content analysis.”
Using large language models to analyze political texts through natural language understanding
Benoit, K. R., de Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 9
Voting, Speechmaking, and the Dimensions of Conict in the US Senate
Kim, I. S. L., Londregan, J., & Ratkovic, M. (2015) · Cited by 6
Should asset managers pay for economic research? A machine learning evaluation
Rybinski, K. I. (2020) · The Journal of Finance and Data Science · doi:10.1016/j.jfds.2020.08.001 · Cited by 6
Nowacki, C., Monk, A., & Decoster, B. (2020) · Environment and Planning A Economy and Space · doi:10.1177/0308518x20951808 · Cited by 6
“These include being careful and transparent about (1) cleaning the texts, (2) choosing the number of topics, (3) naming the topics and (4) validating the results (Grimmer and Stewart, 2013).”
BackgroundMethodInfluentialFrom controversy to common ground
Litofcenko, J., Vogler, A., Meyer, M., & Mehrwald, M. (2023) · Journal of Language and Politics · doi:10.1075/jlp.22124.lit · Cited by 6
Peng, X., Lee, J., Adamow, M. J., Maher, C. A., Postow, M. A., Callahan, M. K., Panageas, K. S., & Shen, R. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2023.100546 · Cited by 5
File, D., File, B., Bőthe, B., Griffiths, M. D., & Demetrovics, Z. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287564 · Cited by 6
More than Meets the Eye? Using Text Analytic Techniques to Unpack School Mission Statements
Bali, V. A., & Higgins, D. (2023) · SAGE Open · doi:10.1177/21582440231218495 · Cited by 6
“In this study we considered measurement validation in two forms: statistical and semantic (Grimmer & Stewart, 2013; Isoaho et al., 2021; Quinn et al., 2010).”
BackgroundMethodExploring public voice on social media: Twitter Users' views on the circular economy
Eid, M. A. H., Dandis, A. O., Cathro, V. I., & Parackal, M. (2024) · Sustainable Development · doi:10.1002/sd.3028 · Cited by 6
Laurer, M., van Atteveldt, W., Casas, A., & Welbers, K. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2024.2378690 · Cited by 12
“Computational social scientists have warned about the challenges of validity and computational text analysis methods for a long time (Grimmer & Stewart, 2013) and recent evidence suggests that there is still a relevant gap between the ambitions of our tools and the validity of their outputs (Baden…”
Smoliński, P. R., Januszewicz, J., & Winiarski, J. (2024) · Proceedings of the International Conference on Information Systems Development · doi:10.62036/isd.2024.95 · Cited by 12
Pipal, C., Schoonvelde, M., Schumacher, G., & Boiten, M. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2024.2383453 · Cited by 9
“However, the importance of methodological validation remains critical (Grimmer & Stewart, 2013), requiring extensive validation by the analyst who cannot escape from close reading of key texts to ensure that model results are indeed reliable and valid.”
Between-Firm Inequality and Informal Social Relations
Wilmers, N., Tong, D., & Zhang, V. (2025) · American Journal of Sociology · doi:10.1086/734909 · Cited by 6
Kaufer, D. S., Ishizaki, S., & Cai, X. (2016) · Journal of Writing Research · doi:10.17239/jowr-2016.07.03.07 · Cited by 7
Multi-word Structural Topic Modelling of ToR Drug Marketplaces
Guarino, S., & Santoro, M. (2018) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2018.00048 · Cited by 6
“Although K has a significant impact on the outcomes of any Topic Modelling implementation, there is no established methodology in the literature to find the “correct” value for K for a given corpus [14].”
BackgroundSon, K. (2020) · Globalization and Health · doi:10.1186/s12992-020-00633-9 · Cited by 6
Психологические причины коррупции: роль тревоги
Миронова, А., & Tatarko, A. N. (2021) · Journal of Economic Sociology · doi:10.17323/1726-3247-2021-1-11-34 · Cited by 7
Navalny’s Digital Dissidents: A New Dataset on a Russian Opposition Movement
Dollbaum, J. M., & Семенов, А. В. (2021) · Problems of Post-Communism · doi:10.1080/10758216.2021.1893123 · Cited by 6
“…into the survey (e.g., on motivations and demands) permits conducting text analysis techniques (sentiment analysis and topic modeling) that combine the flexibility of traditional qualitative methods with the standardization and the large N of quantitative analysis (Grimmer and Stewart 2013).”
MethodShaw, L. (2021) · Socio-Economic Review · doi:10.1093/ser/mwab009 · Cited by 5
Meeks, L. (2021) · Presidential Studies Quarterly · doi:10.1111/psq.12759 · Cited by 6
Stundal, L., Bagozzi, B. E., Freeman, J. R., & Holmes, J. (2021) · Political Analysis · doi:10.1017/pan.2021.40 · Cited by 5
“The former evaluates whether machine and human coders extract the same information from the same text (Grimmer and Stewart 2013, 279).”
BackgroundTeaching Computational Social Science for All
Kim, J. Y., & Ng, Y. M. M. (2022) · PS Political Science & Politics · doi:10.1017/s1049096521001815 · Cited by 6
Political Science and Digitalization – Global Perspectives
Kneuer, M., Milner, H. V., & Budrich, V. B. (2019) · Verlag Barbara Budrich eBooks · doi:10.3224/84742332 · Cited by 12
Cavalieri, A., Ducange, P., Fabi, S., Russo, F., & Tonellotto, N. (2022) · Journal of Information Technology & Politics · doi:10.1080/19331681.2022.2082622 · Cited by 7
“…& Marcelloni, 2015), medicine (Hughes, Li, Kotoulas, & Suzumura, 2017), social media analysis (Yoo, Song, & Jeong, 2018), psychology (Burdisso, Errecalde, & Montes-y G ´omez, 2019), sentiment & opinion identification (Onan, 2018, 2020a, 2020b), and political science (Grimmer & Stewart, 2013).”
BackgroundBig Data Challenges in Social Sciences: An NLP Analysis
Zwilling, M. (2022) · Journal of Computer Information Systems · doi:10.1080/08874417.2022.2085211 · Cited by 6
“21 The title and abstract of each document were also manually read to verify the document’s pertinence to the explored domain.”
MethodInfluentialSigari, S., & Gandomi, A. H. (2022) · Humanities and Social Sciences Communications · doi:10.1057/s41599-022-01325-y · Cited by 6
“First, they create a training set based on features and categories in the training set, and then they apply the learning method to infer the labels for the test set (Grimmer and Stewart, 2013).”
MethodNya ytterhögerpartier i Finland (2000–2022)
Fagerholm, A. (2022) · Politiikka · doi:10.37452/politiikka.117042 · Cited by 6
Hung, S., & Chang, S. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122276 · Cited by 9
“To validate our 4-topic model, we carry out external evaluation by presenting our results in several E/MBA classrooms to secure feedback (DiMaggio et al., 2013; Grimmer and Stewart, 2013).”
Becoming European Through Football Media? Representations of Europe in German Football News Coverage
Biel, J., Finger, T., Reinke, V., et al. (2023) · International Journal of Sport Communication · doi:10.1123/ijsc.2023-0004 · Cited by 6
Moral, P., & Marco, G. (2023) · Communication Research and Practice · doi:10.1080/22041451.2023.2188699 · Cited by 7
Küsters, A. (2023) · Jahrbuch für Wirtschaftsgeschichte / Economic History Yearbook · doi:10.1515/jbwg-2023-0004 · Cited by 7
Polo, F. M., Saxena, S., & Alexopoulos, C. (2023) · International Journal of Public Administration · doi:10.1080/01900692.2023.2199174 · Cited by 9
“It becomes pertinent to cull out interpretations from these texts for deepening our understanding of the politics of the day (Grimmer & Stewart, 2013).”
Metadata
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
