The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
161 Own interpretation · 38 No basis given · 28 Sources not found · 26 No full text of the sources · 21 Own result · 11 Supported · 8 Not checked yet · 1 Partly supported · 1 Own method
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Supportedcites Hand 2006 source
Section 3, “Design Sample Selection,” says the classical supervised-classification paradigm assumes the design set is randomly drawn from the same distribution as future points to be classified. That supports the citing sentence’s application of the assumption to documents; the supplied copy does not provide a page number. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract says that determining in advance which clustering objective function will produce a useful or insightful partition for a given dataset is unknown and difficult, if not logically impossible. It also says there is no method for choosing beforehand which unsupervised approach will yield the most useful clustering for a particular application. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
In Model Output (page unavailable), the paper calls its speech-topic assignments “parameters of direct interest” and uses them to measure attention to political topics over time and by speaker group. Model Specification and Sensitivity Analysis (page unavailable) says the authors selected topic categories to correspond to areas of governmental competence. Together, these support the paraphrase. (LLM, conf=0.87)
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
The model description explains that speech topics are inferred and their probabilities vary by day. The Abstract reports a daily series of attention to topics in U.S. Senate speeches. These passages support the quoted claim; they do not establish the separate 'first topic model' characterization in the citing sentence. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology (§2.4) says the authors project a metric space of clusterings into two Euclidean dimensions and that nearby points represent similar clusterings. Methodology (§2.3) defines similarity by how documents are grouped across clusterings, matching the citing sentence’s paraphrase. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Categorizing Texts: Methods, Assumptions, and Costs” (page not available), the paper says its topic model infers categories rather than requiring them to be fixed beforehand, and shifts effort to postanalysis. “Topic Interpretation and Intratopic Semantic Validity” (page not available) describes examining the resulting topics and documents to assess their validity. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“empirically these sentences are rare.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the restrained category, the recall is 0.75.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 158
- It may state two separate claims as one. 34
Every source the paper cites for this claim was checked and supports it.
In the paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Supportedcites Berinsky, Huber, and Lenz 2012 source
The Introduction says MTurk can facilitate low-cost experiments, and Section 2 describes how researchers post tasks for workers and recruit survey participants. The Abstract also describes MTurk experiments as easy to field. Together, these passages support the claim that the platform can make data acquisition efficient. (LLM, conf=0.91)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Supportedcites Monroe, Colaresi, and Quinn (2008) source
Section 3 describes methods for selecting and evaluating words that distinguish Democratic from Republican Senate speeches. Section 3.1 makes clear that the categories are already identified: the speakers’ partisanship is observed, not something the methods need to infer. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Grimmer and King 2011 source
Section 3.1 (page unavailable) defines cluster quality from human judgments as within-cluster similarity minus between-cluster similarity, and Section 3 reports using that measure to compare clusterings. The measure is comparative rather than an absolute test of semantic validity, consistent with the paper’s caution that there is no reliable gold-standard clustering for this purpose. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract (p. 2643) says the authors develop a space containing existing clustering methods and millions of additional solutions based on combinations of existing clusterings, then enable users to explore it quickly. Section 2, Methodology, describes how users navigate that space to select useful clusterings. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology section describes CAC as organizing a large number of possible clusterings in a visual space so researchers can efficiently select useful ones. The Abstract likewise says users can explore a space containing existing methods and millions of additional solutions and quickly identify useful clusterings. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Exogenous Events and Predictive Validity,” the paper defines predictive validity as correspondence between a measure and external events, then links spikes in topic attention to events including 9/11 and debates over military action. The Discussion says examining relationships with such events helps assess the validity of its attention measures. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The topics in Table 4 exemplify the utility of unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language is the medium for politics and political conflict.”
- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Politics and political conflict often occur in the written and spoken word.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential when applying automated content methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it substantially reduces the costs of analyzing large collections of text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“to understand what politics is about we need to know what political actors are saying and writing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“require extensive and problem-specific validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they are no substitute for careful thought and close reading”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of any one classifier can vary substantially across contexts”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“And hiring coders to manually read all documents is still very expensive.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One way to improve upon dictionaries are supervised methods”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“automated content methods are incorrect models of language.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding the separating words is also relatively easy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tone is just one type of analysis a dictionary method can perform.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is often difficult, and sometimes impossible, to validate the output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe described above is one way to represent text as data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather than replace humans, computers amplify human abilities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is also substantial variation within families of models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validations are necessary to establish its meaning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- The authors state this as an assumption: Ideology dominates the language used in the text.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The data generation process for any text (including this one) is a mystery-even to linguists.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of applied research uses stemming.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“sometimes the problem calls for a completely different approach.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stemmer is a much cruder algorithm, but considerably faster.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance does not seem to matter for most applications”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“automated scraping methods make acquiring these data easier (Jackman 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These programs simply provide the researcher with output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying FAC algorithms to texts may have mixed results”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
What the sources say
- Not checked — no full text of the sourcecites Spirling (2012) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Eggers and Hainmueller 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated methods also rely on having a sufficient number of words to use reliably.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Porter 1980 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Not matchedcites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these require the close reading of texts and thoughtful analysis by the researcher.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assigning texts to categories is the most common use of content analysis methods in political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“not all of language's complexity is necessary to effectively analyze texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
What the sources say
- Not matchedcites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.3
What the sources say
- Not checked — no full text of the sourcecites van der Vaart, Dudoit, and van der Laan 2006 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Not matchedcites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
- Not checked — no full text of the sourcecites Hastie, Tibshirani, and Friedman 2001 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Dictionary methods require scholars to identify words that separate classes beforehand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number necessary will depend upon the specific application of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
- Not checked — no full text of the sourcecites Hillard, Purpura, and Wilkerson 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAC methods are useful, but are necessarily model dependent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1 FAC
What the sources say
- Not matchedcites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- The authors state this as an assumption: Given a document's category, the words are generated independently.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Supervised and dictionary methods assume a well-defined set of categories.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Yiannakis 1982 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the K-means' objective function is multimodal and non-continuous.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not checked — no full text of the sourcecites Blei 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and therefore several random restarts are often necessary to find the global optimum.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § 6.1.2 Mixed membership models · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The goal is revelation of substantively interesting information.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To estimate a topic, the models use the co-occurrence of words across documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Situating the models in a data-generating process makes including additional structure straightforward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preprocessed texts represent a substantial simplification of the documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CAC methods provide a way to explore these different assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, CAC provides an accelerated way to explore new collections of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many other distance metrics are possible, and each will lead to different clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But CAC does place substantial burden on users to identify clusterings that are useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.2 Unsupervised Methods for Scaling
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Lowe 2008 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But nearly all political actors speak.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We caution that this validation does not obviate the need for unsupervised methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that any one method's performance will be context specific.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimating locations using existing data is often difficult and sometimes impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not matchedcites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Experimental — accuracy not yet measuredIn the citing paper
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
In the citing paper
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 25,941
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
(1967) · Cited by 23,719
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Items 301–400 of 3,308
Climate risk and IMF surveillance policy: a baseline analysis
Ramos, L. S., Gallagher, K. P., Stephenson, C., & Monasterolo, I. (2021) · Climate Policy · doi:10.1080/14693062.2021.2016363 · Cited by 64
“The IMF also models spillover propagation channels because of a significant increase in global carbon prices that impact the Norwegian oil sector (Grippa & Mann, 2020).”
BackgroundAceves, P., & Evans, J. A. (2023) · Organization Science · doi:10.1287/orsc.2023.1686 · Cited by 74
Thematic analysis in the area of education: a practical guide
Kushnir, I. (2025) · Cogent Education · doi:10.1080/2331186x.2025.2471645 · Cited by 111
Levy, K. E. C., & Franklin, M. (2013) · Social Science Computer Review · doi:10.1177/0894439313506847 · Cited by 64
“This approach priori-tizes the utility of the model as a means for discovery in place of statistical measures of model fit, which may be less semantically meaningful (Chang, Boyd-Graber, Gerrish, Wang, & Blei, 2009; Grimmer & Stewart, 2013).”
BackgroundMethodInfluentialCommon Core State Standards on Twitter: Public Sentiment and Opinion Leaders
Wang, Y., & Fikis, D. J. (2017) · Educational Policy · doi:10.1177/0895904817723739 · Cited by 61
“Rather, the automated analytical approaches compliment the conventional approaches to analyze qualitative data (Grimmer & Stewart, 2013; King, 2011, 2016).”
BackgroundMethodMedia coverage and immigration worries: Econometric evidence
Benesch, C., Loretz, S., Stadelmann, D., & Thomas, T. (2019) · Journal of Economic Behavior and Organization · doi:10.1016/j.jebo.2019.02.011 · Cited by 91
Decision Trees and Random Forests: Machine Learning Techniques to Classify Rare Events
Hegelich, S. (2016) · European Policy Analysis · doi:10.18278/epa.2.1.7 · Cited by 63
Communicating with Warmth in Distributive Negotiations Is Surprisingly Counterproductive
Jeong, M., Minson, J. A., Yeomans, M., & Francesca Gino, B. (2019) · Management Science · doi:10.1287/mnsc.2018.3199 · Cited by 63
“Like any open-ended text data, these messages varied along many dimensions (Jurafsky and Martin 2009, Grimmer and Stewart 2013).”
BackgroundDeveloping Theory Through Integrating Human and Machine Pattern Recognition
", & Lindberg, A. (2020) · Journal of the Association for Information Systems · doi:10.17705/1jais.00593 · Cited by 80
“These opportunities allow qualitative researchers to expand but not necessarily replace their current toolboxes (Grimmer & Stewart, 2013).”
BackgroundMethodInfluentialTopic Modeling: Perspectives From a Literature Review
Aguirre, A. M. G., Robledo, S., & Zuluaga, M. (2022) · IEEE Access · doi:10.1109/access.2022.3232939 · Cited by 94
“For example, Greene and Cross [39] studied a political agenda using TM, and Grimmer and Stewart [40] presented the advantages and disadvantages of using this method in political texts.”
MethodSeeded Sequential LDA: A Semi-Supervised Algorithm for Topic-Specific Analysis of Sentences
Watanabe, K., & Baturo, A. (2023) · Social Science Computer Review · doi:10.1177/08944393231178605 · Cited by 77
“Among several topic models, Latent Dirichlet allocation (LDA) (Blei et al., 2003) has been arguably the most popular algorithm in social science research (Grimmer & Stewart, 2013).”
Facing Big Data: Making sociology relevant
Mützel, S. (2015) · Big Data & Society · doi:10.1177/2053951715599179 · Cited by 57
“…2009; Blei et al., 2003), topic modeling has recently received heightened attention in the humanities (e.g. Jockers, 2013; Meeks and Weingart, 2013; Moretti, 2013) and the social sciences (e.g. Grimmer and Stewart, 2013; Kaplan and Vakili, 2014; Mohr and Bogdanov, 2013; Ramage et al., 2009).”
BackgroundSpratt, T., Devaney, J., & Hayes, D. (2015) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2015.01.015 · Cited by 62
Burge, R. P., & Williams, M. D. (2019) · Journal of Religion Media and Digital Culture · doi:10.1163/21659214-00803004 · Cited by 62
“Words need to be understood in the context of sentences/tweets to understand if they are being used in a similar way (Grimmer & Stewart, 2013).”
MethodAn Agenda-Setting Theory of Electoral Competition
Dragu, T., & Fan, X. (2016) · The Journal of Politics · doi:10.1086/686310 · Cited by 56
Brouwer, J., van der Woude, M., & van der Leun, J. (2017) · European Journal of Criminology · doi:10.1177/1477370816640136 · Cited by 64
“Of course, as Tolley (2015: 968) also acknowledges, the distinction between manual and automated analyses is far from clear-cut, with computers amplifying rather than replacing human work (see also Grimmer and Stewart, 2013).”
BackgroundLee, Y.-L., Jung, M.-J., Nathan, R., & Chung, J. (2020) · Sustainability · doi:10.3390/su12156072 · Cited by 67
Simulating pluralism: the language of democracy in hegemonic authoritarianism
Maerz, S. F. (2019) · Political Research Exchange · doi:10.1080/2474736x.2019.1605834 · Cited by 65
“The recently expanded toolbox of quantitative text analysis provides innovative approaches for analysing political texts (Grimmer and Stewart 2013) – yet, this burgeoning literature has been mainly focused on democratic settings.”
BackgroundNelimarkka, M., Leinonen, T., Durall, E., & Dean, P. (2021) · Computers & Education · doi:10.1016/j.compedu.2021.104269 · Cited by 61
Computational grounded theory revisited: From computer-led to computer-assisted text analysis
Carlsen, H. B., & Ralund, S. (2022) · Big Data & Society · doi:10.1177/20539517221080146 · Cited by 61
“Two approaches to computational text analysis are commonly juxtaposed in an attempt to overcome the tasks iden-ti fi ed above: a supervised approach and an unsupervised approach (Grimmer and Stewart, 2013; Evans and Aceves, 2016; Nelson et al., 2018; DiMaggio, 2015).”
MethodAll without leaving home: building a conceptual model of virtual tourism experiences
Wei, W., Baker, M. A., & Önder, İ. (2022) · International Journal of Contemporary Hospitality Management · doi:10.1108/ijchm-12-2021-1560 · Cited by 62
Gençoglu, B., Helms‐Lorenz, M., Maulana, R., Jansen, E. P. W. A., & Gencoglu, O. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104682 · Cited by 75
Economic history goes digital: topic modeling the Journal of Economic History
Wehrheim, L. (2018) · Cliometrica · doi:10.1007/s11698-018-0171-7 · Cited by 64
Circular Economy in the Triple Helix of Innovation Systems
Anttonen, M., Lammi, M., Mykkänen, J., & Repo, P. (2018) · Sustainability · doi:10.3390/su10082646 · Cited by 51
“In the context of social sciences, it has been used, for example, to analyse political frames held by decision-makers and media organisations [46,47], and for discovering similarities and differences of political logics in social movements [45].”
BackgroundAnalysis of the style and the rhetoric of the 2016 US presidential primaries
Savoy, J. (2017) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqx007 · Cited by 60
“A more general overview of using different computer-based strategy to detect and extract topical information from political texts can be found in (Grimmer and Stewart, 2013).”
BackgroundAguilar, S. J., & Baek, C. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230312 · Cited by 61
“This approach was the most appropriate, given that we do not know of a collection of text responses that we could use as a reference category to use more sophisticated methods [50].”
MethodSong, H., Eberl, J., & Eisele, O. (2020) · Journal of Communication · doi:10.1093/joc/jqaa009 · Cited by 56
MPEDS: Automating the Generation of Protest Event DataPreprint
Hanna, A. (2017) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/xuqmv · Cited by 61
Eggers, A. C., & Spirling, A. (2014) · American Journal of Political Science · doi:10.1111/ajps.12090 · Cited by 59
A concrete example of construct construction in natural language
Yeomans, M. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.10.008 · Cited by 61
“We created a large list of features, using a relatively simple bag-of-ngrams approach (Grimmer & Stewart, 2013; Hirschberg & Manning, 2015; Jurafsky & Martin, 2019).”
BackgroundMethodMamo, Y., Crompton, H., Burke, D., & Nickel, C. (2024) · TechTrends · doi:10.1007/s11528-024-00954-1 · Cited by 65
“It is important to note that any type of quantitative model of languages is not globally the best method and needs to be validated (Grimmer & Stewart, 2013).”
Niculae, V., Suen, C., Zhang, J., Danescu-Niculescu-Mizil, C., & Leskovec, J. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741688 · Cited by 55
“Furthermore, natural language techniques can be better tuned to insights from political science in order to produce tools and resources more suited for analyzing political discourse [10].”
BackgroundContent Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent Constructs
Lind, F., Gruber, M., & Boomgaarden, H. G. (2017) · Communication Methods and Measures · doi:10.1080/19312458.2017.1317338 · Cited by 57
“Methodological innovation in content analysis procedures rather focuses on automated, computerassisted techniques (e.g., Boumans & Trilling, 2016; Grimmer & Stewart, 2013; Jacobi, van Atteveldt, & Welbers, 2016).”
MethodExamining multiplicity and dynamics of publics’ crisis narratives with large-scale Twitter data
Zhao, X., Zhan, M. M., & Jie, C. (2018) · Public Relations Review · doi:10.1016/j.pubrev.2018.07.004 · Cited by 69
Temporal topic modeling applied to aviation safety reports: A subject matter expert review
Robinson, S. D. (2019) · Safety Science · doi:10.1016/j.ssci.2019.03.014 · Cited by 67
Position Taking or Position Ducking? A Theory of Public Debate in Single-Party Legislatures
Schuler, P. (2018) · Comparative Political Studies · doi:10.1177/0010414018758765 · Cited by 58
“In terms of validation, one of the perils of text analysis is that text models are inherently “wrong” (Grimmer & Stewart, 2013).”
BackgroundMethodResearcher reasoning meets computational capacity: Machine learning for social science
Lundberg, I., Brand, J. E., & Jeon, N. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102807 · Cited by 63
Planning prompts increase and forecast course completion in massive open online courses
Yeomans, M., & Reich, J. (2017) · Proceedings of the Seventh International Learning Analytics & Knowledge Conference · doi:10.1145/3027385.3027416 · Cited by 61
“Like most text data, the content of the plans are unstructured and high-dimensional, which poses problems for traditional analytic approaches [13, 17, 32].”
BackgroundHow to Make Causal Inferences Using TextsPreprint
Egami, N., Fong, C., Grimmer, J., Roberts, M. E., & Stewart, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.02163 · Cited by 53
“While validation approaches may vary—this necessarily involves reading documents ( 15 , 22 , 29 )—it is helpful to fix a set of human-assigned labels, example documents, and automated keyword labels in advance to avoid subtle influence from the test set.”
Self-citationHow African countries respond to fake news and hate speech
Garbe, L., Selvik, L.-M., & Lemaire, P. (2023) · Information, Communication & Society · doi:10.1080/1369118x.2021.1994623 · Cited by 57
“…relies on the probabilistic topic model technique Latent Dirichlet Allocation (see Blei, 2012), a technique which uses word counts and not the order of words, it is up to the researcher to infer meaning from the words and topics that appear, rather than assert it (Grimmer & Stewart, 2013, p. 272).”
MethodAlschner, W., Elsig, M., & Polanco, R. (2020) · World Trade Review · doi:10.1017/s147474562000035x · Cited by 57
“Moreover, EDIT is designed to facilitate scholarship and innovative research, in particular the computational analysis of IIAs that treats these texts as data (Gentzkow et al., 2017; Grimmer and Stewart, 2013).”
BackgroundGrossoni, I., Hughes, P., Bezin, Y., et al. (2020) · Engineering Failure Analysis · doi:10.1016/j.engfailanal.2020.104987 · Cited by 57
Müller-Hansen, F., Callaghan, M., Lee, Y. T., Leipprand, A., Flachsland, C., & Minx, J. C. (2021) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101869 · Cited by 56
How do contract performance rates affect entrepreneurs’ risk-averse attitudes? Evidence from China
Sun, Z., Lu, S., Huang, M., Zhuang, J., Lucero, A. M. V., & Osei, C. D. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1112344 · Cited by 59
“, 2020) to incorporate more dimensions into the existing evaluation system and assign weights to secondary indicators through text analysis (Grimmer and Stewart, 2013).”
BackgroundPublic Frame Building: The Role of Source Usage in Times of Crisis
van der Meer, T. G. L. A. (2016) · Communication Research · doi:10.1177/0093650216644027 · Cited by 57
“clustering (Grimmer & Stewart, 2013).”
BackgroundInfluentialDigitization of the Canadian Parliamentary Debates
Beelen, K., Thijm, T. A., Cochrane, C., Halvemaan, K., Hirst, G., Kimmins, M., Lijbrink, S., Marx, M., Naderi, N., Rheault, L., et al. (2017) · Canadian Journal of Political Science · doi:10.1017/s0008423916001165 · Cited by 55
Mourtgos, S. M., & Adams, I. T. (2019) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2019.101627 · Cited by 57
Male and female politicians on Twitter: A machine learning approach
Beltran, J., Gallego, A., Huidobro, A., Romero, E. E., & Padró, L. (2020) · European Journal of Political Research · doi:10.1111/1475-6765.12392 · Cited by 66
Antons, D., Joshi, A. M., & Salge, T. O. (2018) · Journal of Management · doi:10.1177/0149206318774619 · Cited by 63
China's Newsmakers: Official Media Coverage and Political Shifts in the Xi Jinping Era
Jaros, K. A., & Pan, J. (2017) · The China Quarterly · doi:10.1017/s0305741017001679 · Cited by 56
Drivers of the disclosed “connectivity of the capitals”: evidence from integrated reports
Graßmann, M., Fuhrmann, S., & Guenther, T. W. (2019) · Sustainability Accounting Management and Policy Journal · doi:10.1108/sampj-03-2018-0086 · Cited by 59
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Handler, A., Denny, M. J., Wallach, H., & O'Connor, B. (2016) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/w16-5615 · Cited by 54
“Many political scientists have studied the relationship between language usage and party affiliation (Laver et al., 2003; Monroe et al., 2008; Slapin and Proksch, 2008; Quinn et al., 2010; Grimmer and Stewart, 2013).”
BackgroundFraming Immigration in the Canadian and British News Media
Lawlor, A. (2015) · Canadian Journal of Political Science · doi:10.1017/s0008423915000499 · Cited by 64
Quantifying with words: An investigation of the validity of narrative‐derived performance scores
Speer, A. B. (2018) · Personnel Psychology · doi:10.1111/peps.12263 · Cited by 62
Document-based topic coherence measures for news media text
Korenčić, D., Ristov, S., & Šnajder, J. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.07.063 · Cited by 59
Computational Methods in Legal Analysis
Frankenreiter, J., & Livermore, M. A. (2020) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-052720-121843 · Cited by 65
Motolinia, L. (2020) · American Political Science Review · doi:10.1017/s0003055420000672 · Cited by 55
“According to Grimmer and Stewart (2013), the substance of topics is determined by reading the words and documents with a high probability of belonging to each topic and using relevant information of the documents to ensure coherence.”
BackgroundThe Effect of Gender on Interruptions at Congressional Hearings
Miller, M. G., & Sutherland, J. L. (2022) · American Political Science Review · doi:10.1017/s0003055422000260 · Cited by 36
“Our approach resulted in 3,790 hearings—15.8% of the total hearings—coded as 11 We queried the lemmatized hearing title and keyword text to ensure conceptual comparability (Grimmer and Stewart 2013).”
BackgroundInfluentialElevated threat levels and decreased expectations: How democracy handles terrorist threats
Bonilla, T., & Grimmer, J. (2013) · Poetics · doi:10.1016/j.poetic.2013.06.003 · Cited by 54
Self-citationStock market returns and the content of annual report narratives
Wisniewski, T. P., & Yekini, L. S. (2015) · Accounting Forum · doi:10.1016/j.accfor.2015.09.001 · Cited by 62
Investment Arbitration and State-Driven Reform
Alschner, W. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780197644386.001.0001 · Cited by 58
Paradigm Wars Revisited: A Cartography of Graduate Research in the Field of Education (1980–2010)
Galvez, S. M.-N., Heiberger, R. H., & McFarland, D. A. (2019) · American Educational Research Journal · doi:10.3102/0002831219860511 · Cited by 54
“For example, previous research has used STMs to obtain topics that are sensitive to extratextual author characteristics (Grimmer & Stewart, 2013) such as language (Munksgaard & Demant, 2016), nationality (C. Lucas et al., 2015), or source of funding (Farrel, 2016).”
MethodThe Datafication of Hate: Expectations and Challenges in Automated Hate Speech Monitoring
Laaksonen, S.-M., Haapoja, J., Kinnunen, T., Nelimarkka, M., & Pöyhtäri, R. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00003 · Cited by 71
“An iterative model could also solve some problems related to the known issue of context associated with machine learning models: developed models do not perform well if used in another, even slightly different, setting (Yu et al., 2008; Grimmer and Stewart, 2013).”
BackgroundMethodThe Effects of Female Leadership on Women's Voice in Political Debate
Blumenau, J. (2019) · British Journal of Political Science · doi:10.1017/s0007123419000334 · Cited by 59
Theoretical Foundations for Digital Text Analysis
Ignatow, G. (2015) · Journal for the Theory of Social Behaviour · doi:10.1111/jtsb.12086 · Cited by 52
“…set, application of the learning method which involves identifying relationships between features and categories in the training set and then using these to infer labels in the test set, and validating the model output and classifying remaining documents (see Grimmer & Stewart, 2013, pp. 9–10).”
MethodDiscovery of Treatments from Text Corpora
Fong, C., & Grimmer, J. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1151 · Cited by 49
“We selected ten treatments using a combination of guidance from the sIBP, assessment using CE scores, and our own qualitative assessment of the models (Grimmer and Stewart, 2013).”
MethodSelf-citationSelection of the most relevant terms based on a max-min ratio metric for text classification
Rehman, A., Javed, K., Babri, H. A., & Asim, M. N. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.07.028 · Cited by 54
Why are the affluent better represented around the world?
Lupu, N., & Warner, Z. (2021) · European Journal of Political Research · doi:10.1111/1475-6765.12440 · Cited by 53
Rothut, S., Schulze, H., Hohner, J., & Rieger, D. (2023) · New Media & Society · doi:10.1177/14614448231164409 · Cited by 53
“We preprocessed the posts following Grimmer and Stewart (2013): We only included posts containing at least three words; removed punctuation, symbols, numbers, URLs, and stopwords; tokenized the posts; and reduced all terms to their stem using the quanteda-package (Benoit et al., 2018).”
Hokka, J., & Nelimarkka, M. (2019) · New Media & Society · doi:10.1177/1461444819868686 · Cited by 52
“Applying visual big data analysis was hindered by validity and contextualisation challenges common with text data or any form of big data (e.g. boyd and Crawford, 2012; Grimmer and Stewart, 2013).”
MethodAutomatic Detection of Comment Propaganda in Chinese MediaPreprint
Miller, B. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2738325 · Cited by 52
Text‐analysis reveals taxonomic and geographic disparities in animal pollination literature
Millard, J. W., Freeman, R., & Newbold, T. (2019) · Ecography · doi:10.1111/ecog.04532 · Cited by 55
“Given the very large numbers of published papers containing potentially useful information, such technologies are invaluable in automatically drawing together results across lots of studies (Grimmer and Stewart 2013), thereby reducing the duration of the ‘synthesis gap’, or in other words the lag between the practice…”
BackgroundEngstrom, T., Strong, J., Sullivan, C. M., & Pole, J. D. (2022) · International Journal of Qualitative Methods · doi:10.1177/16094069221118993 · Cited by 51
“These semi-automated content analysis (SACA) tools can analyse larger volumes of data more ef fi ciently than manual content analysis (MCA), reduce the human bias and subjectivity inherent in MCA and improve reproducibility (Grimmer & Stewart, 2013).”
BackgroundMethodE-petition popularity: Do linguistic and semantic factors matter?
Hagen, L., Harrison, T. M., Uzuner, Ö., May, W., Fake, T., & Katragadda, S. (2016) · Government Information Quarterly · doi:10.1016/j.giq.2016.07.006 · Cited by 57
Narrative as a Method for Eliciting Tacit Knowledge of Climate Variability in Bangladesh
Bremer, S., Blanchard, A., Mamnun, N., Stiller-Reeve, M. A., Haque, M. M., & Tvinnereim, E. M. (2017) · Weather Climate and Society · doi:10.1175/wcas-d-17-0007.1 · Cited by 50
“The fourth theme invited quantitative text analysis methods (Grimmer and Stewart 2013; Tvinnereim and Fl ø ttum 2015), including concordance analysis and bigram frequency counts. a. Weather phenomena and impacts that are important to the community Interview narratives recounted interviewees’ unique…”
Content Analysis: Frequency Distribution of Words
Dicle, M. F., & Dicle, B. (2018) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x1801800205 · Cited by 51
The Solutionist Ethic and the Spirit of Digital CapitalismPreprint
Nachtwey, O., & Seidl, T. (2020) · doi:10.31235/osf.io/sgjzq · Cited by 51
Challenges in researching terrorism from the field
Scott Atran, B., Axelrod, R., DAVIS, R. E., & Fischhoff, B. (2017) · Science · doi:10.1126/science.aaj2037 · Cited by 50
Ferrara, F. M., Masciandaro, D., Moschella, M., et al. (2021) · European Journal of Political Economy · doi:10.1016/j.ejpoleco.2021.102143 · Cited by 50
Computational Text Analysis for Social Science: Model Assumptions and Complexity
O'Connor, B., Bamman, D., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473291 · Cited by 48
TopicCheck: Interactive Alignment for Assessing Topic Model Stability
Chuang, J., Roberts, M. E., Stewart, B., Weiss, R., Tingley, D. H., Grimmer, J., & Heer, J. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1018 · Cited by 50
“In a comprehensive survey of automatic content analysis methods, Grimmer et al. (2011) highlight the need to validate models through close reading and model comparison, and advise against the use of software that “ simply provide the researcher with output ” with no capability to ensure the output…”
BackgroundMethodSelf-citationMicroblogging as an extension of science reporting
Büchi, M. (2016) · Public Understanding of Science · doi:10.1177/0963662516657794 · Cited by 60
“This topic modeling approach differs substantially from communication research’s traditional content analyses in that it is automated and unsupervised (Grimmer and Stewart, 2013): There is no training data, no human coding, and no codebook.”
Syed, S., Borit, M., & Spruit, M. R. (2018) · Fish and Fisheries · doi:10.1111/faf.12280 · Cited by 50
“…modelling approaches have been helpful in elucidating the key ideas within a set of documents, such as articles published in the journal PNAS (Griffiths & Steyvers, 2004), political science texts (Grimmer & Stewart, 2013) or data- driven journalism (Rusch, Hofmarcher, Hatzinger, & Hornik, 2013).”
BackgroundMethodRefugee debate and networked framing in the hybrid media environment
Pöyhtäri, R., Nelimarkka, M., Nikunen, K., Ojala, M., Pantti, M., & Pääkkönen, J. (2019) · International Communication Gazette · doi:10.1177/1748048519883520 · Cited by 50
“A second labelling was based on the selection of 20 random text examples from 200 documents with the highest posterior values to increase validity (Grimmer and Stewart, 2013).”
MethodComputer-Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses
Reich, J., Tingley, D. H., Leder-Luis, J., Roberts, M. E., & Stewart, B. (2014) · Journal of Learning Analytics · doi:10.18608/jla.2015.21.8 · Cited by 72
Self-citationTextual Predictors of Bill Survival in Congressional Committees
Yano, T., Smith, N. A., & Wilkerson, J. (2018) · doi:10.1184/r1/6473729 · Cited by 48
Rauh, C., & Schwalbach, J. (2020) · doi:10.31235/osf.io/cd2qs · Cited by 50
Cognitive Biases and Communication Strength in Social Networks: The Case of Episodic Frames
Aarøe, L., & Petersen, M. B. (2018) · British Journal of Political Science · doi:10.1017/s0007123418000273 · Cited by 52
“Finally, all of the key analyses reported below were replicated using dictionaries in which all of the words had been stemmed (Grimmer and Stewart 2013, 272).”
MethodTracking the Coverage of Public Policy in Mass Media
Soroka, S., & Wlezien, C. B. (2018) · Policy Studies Journal · doi:10.1111/psj.12285 · Cited by 51
Ringe, N. (2021) · University of Michigan Press eBooks · doi:10.3998/mpub.12080141 · Cited by 51
Unpacking the black box of ICO white papers: A topic modeling approach
Thewissen, J., Shrestha, P., Torsin, W., & Pastwa, A. M. (2022) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2022.102225 · Cited by 50
Hilbert, M., Ahmed, S., Cho, J., Liu, B., & Luu, J. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1479843 · Cited by 49
Adapting or Freezing? Ideological Reactions of Communist Regimes to a Post-Communist World
Dukalskis, A., & Gerschewski, J. (2018) · Government and Opposition · doi:10.1017/gov.2018.40 · Cited by 49
Identifying Potential Campaign Themes to Prevent Youth Initiation of E-Cigarettes
Sangalang, A., Volinsky, A. C., Liu, J., Yang, Q., Lee, S. J., Gibson, L., & Hornik, R. (2019) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2018.07.039 · Cited by 49
Improving public services by mining citizen feedback: An application of natural language processing
Kowalski, R., Estève, M., & Mikhaylov, S. (2020) · Public Administration · doi:10.1111/padm.12656 · Cited by 71
“Online written reviews can be validated against state-of-the-art structured forms of user feedback, such as carefully drafted surveys involving large numbers of reviewers (Grimmer and Stewart 2013; Rogge et al. 2017).”
BackgroundMethodInfluentialMedia Coverage of Campaign Promises Throughout the Electoral Cycle
Müller, S. (2020) · Political Communication · doi:10.1080/10584609.2020.1744779 · Cited by 52
“I opt for a combination of human judgment and quantitative text analysis to filter relevant sentences from the corpus (Grimmer & Stewart 2013; Soroka & Wlezien 2019).”
MethodRe-engaging with Sustainability in the Anthropocene Era
Hoffman, A. J., & Jennings, P. D. (2018) · Cambridge University Press eBooks · doi:10.1017/9781108628068 · Cited by 55
Social Media Sellout - The Increasing Role of Product Promotion on YouTubePreprint
Schwemmer, C., & Ziewiecki, S. (2018) · doi:10.31235/osf.io/tskjb · Cited by 49
Cross-Domain Topic Classification for Political Texts
Osnabrügge, M., Ash, E., & Morelli, M. (2021) · Political Analysis · doi:10.1017/pan.2021.37 · Cited by 48
“Meanwhile, an increasing number of studies are using a hand-codedsubsampleoftextdatatoautomaticallycodeunlabeleddocumentsusingsupervised-learning techniques (Grimmer and Stewart 2013).”
Differentiation in microenterprises
Carlson, N. A. (2022) · Strategic Management Journal · doi:10.1002/smj.3463 · Cited by 54
“There exists a growing tradition of computational approaches to text in the social sciences (Teodorescu, 2017; Grimmer and Stewart, 2013; Gentzkow et al., 2019).”
Background
Metadata
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
