The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
161 Own interpretation · 38 No basis given · 28 Sources not found · 26 No full text of the sources · 21 Own result · 11 Supported · 8 Not checked yet · 1 Partly supported · 1 Own method
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Supportedcites Hand 2006 source
Section 3, “Design Sample Selection,” says the classical supervised-classification paradigm assumes the design set is randomly drawn from the same distribution as future points to be classified. That supports the citing sentence’s application of the assumption to documents; the supplied copy does not provide a page number. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract says that determining in advance which clustering objective function will produce a useful or insightful partition for a given dataset is unknown and difficult, if not logically impossible. It also says there is no method for choosing beforehand which unsupervised approach will yield the most useful clustering for a particular application. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
In Model Output (page unavailable), the paper calls its speech-topic assignments “parameters of direct interest” and uses them to measure attention to political topics over time and by speaker group. Model Specification and Sensitivity Analysis (page unavailable) says the authors selected topic categories to correspond to areas of governmental competence. Together, these support the paraphrase. (LLM, conf=0.87)
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
The model description explains that speech topics are inferred and their probabilities vary by day. The Abstract reports a daily series of attention to topics in U.S. Senate speeches. These passages support the quoted claim; they do not establish the separate 'first topic model' characterization in the citing sentence. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology (§2.4) says the authors project a metric space of clusterings into two Euclidean dimensions and that nearby points represent similar clusterings. Methodology (§2.3) defines similarity by how documents are grouped across clusterings, matching the citing sentence’s paraphrase. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Categorizing Texts: Methods, Assumptions, and Costs” (page not available), the paper says its topic model infers categories rather than requiring them to be fixed beforehand, and shifts effort to postanalysis. “Topic Interpretation and Intratopic Semantic Validity” (page not available) describes examining the resulting topics and documents to assess their validity. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“empirically these sentences are rare.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the restrained category, the recall is 0.75.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 158
- It may state two separate claims as one. 34
Every source the paper cites for this claim was checked and supports it.
In the paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Supportedcites Berinsky, Huber, and Lenz 2012 source
The Introduction says MTurk can facilitate low-cost experiments, and Section 2 describes how researchers post tasks for workers and recruit survey participants. The Abstract also describes MTurk experiments as easy to field. Together, these passages support the claim that the platform can make data acquisition efficient. (LLM, conf=0.91)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Supportedcites Monroe, Colaresi, and Quinn (2008) source
Section 3 describes methods for selecting and evaluating words that distinguish Democratic from Republican Senate speeches. Section 3.1 makes clear that the categories are already identified: the speakers’ partisanship is observed, not something the methods need to infer. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Grimmer and King 2011 source
Section 3.1 (page unavailable) defines cluster quality from human judgments as within-cluster similarity minus between-cluster similarity, and Section 3 reports using that measure to compare clusterings. The measure is comparative rather than an absolute test of semantic validity, consistent with the paper’s caution that there is no reliable gold-standard clustering for this purpose. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract (p. 2643) says the authors develop a space containing existing clustering methods and millions of additional solutions based on combinations of existing clusterings, then enable users to explore it quickly. Section 2, Methodology, describes how users navigate that space to select useful clusterings. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology section describes CAC as organizing a large number of possible clusterings in a visual space so researchers can efficiently select useful ones. The Abstract likewise says users can explore a space containing existing methods and millions of additional solutions and quickly identify useful clusterings. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Exogenous Events and Predictive Validity,” the paper defines predictive validity as correspondence between a measure and external events, then links spikes in topic attention to events including 9/11 and debates over military action. The Discussion says examining relationships with such events helps assess the validity of its attention measures. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The topics in Table 4 exemplify the utility of unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language is the medium for politics and political conflict.”
- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Politics and political conflict often occur in the written and spoken word.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential when applying automated content methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it substantially reduces the costs of analyzing large collections of text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“to understand what politics is about we need to know what political actors are saying and writing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“require extensive and problem-specific validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they are no substitute for careful thought and close reading”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of any one classifier can vary substantially across contexts”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“And hiring coders to manually read all documents is still very expensive.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One way to improve upon dictionaries are supervised methods”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“automated content methods are incorrect models of language.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding the separating words is also relatively easy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tone is just one type of analysis a dictionary method can perform.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is often difficult, and sometimes impossible, to validate the output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe described above is one way to represent text as data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather than replace humans, computers amplify human abilities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is also substantial variation within families of models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validations are necessary to establish its meaning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- The authors state this as an assumption: Ideology dominates the language used in the text.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The data generation process for any text (including this one) is a mystery-even to linguists.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of applied research uses stemming.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“sometimes the problem calls for a completely different approach.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stemmer is a much cruder algorithm, but considerably faster.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance does not seem to matter for most applications”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“automated scraping methods make acquiring these data easier (Jackman 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These programs simply provide the researcher with output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying FAC algorithms to texts may have mixed results”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
What the sources say
- Not checked — no full text of the sourcecites Spirling (2012) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Eggers and Hainmueller 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated methods also rely on having a sufficient number of words to use reliably.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Porter 1980 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Not matchedcites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these require the close reading of texts and thoughtful analysis by the researcher.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assigning texts to categories is the most common use of content analysis methods in political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“not all of language's complexity is necessary to effectively analyze texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
What the sources say
- Not matchedcites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.3
What the sources say
- Not checked — no full text of the sourcecites van der Vaart, Dudoit, and van der Laan 2006 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Not matchedcites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
- Not checked — no full text of the sourcecites Hastie, Tibshirani, and Friedman 2001 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Dictionary methods require scholars to identify words that separate classes beforehand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number necessary will depend upon the specific application of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
- Not checked — no full text of the sourcecites Hillard, Purpura, and Wilkerson 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAC methods are useful, but are necessarily model dependent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1 FAC
What the sources say
- Not matchedcites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- The authors state this as an assumption: Given a document's category, the words are generated independently.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Supervised and dictionary methods assume a well-defined set of categories.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Yiannakis 1982 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the K-means' objective function is multimodal and non-continuous.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not checked — no full text of the sourcecites Blei 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and therefore several random restarts are often necessary to find the global optimum.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § 6.1.2 Mixed membership models · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The goal is revelation of substantively interesting information.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To estimate a topic, the models use the co-occurrence of words across documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Situating the models in a data-generating process makes including additional structure straightforward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preprocessed texts represent a substantial simplification of the documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CAC methods provide a way to explore these different assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, CAC provides an accelerated way to explore new collections of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many other distance metrics are possible, and each will lead to different clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But CAC does place substantial burden on users to identify clusterings that are useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.2 Unsupervised Methods for Scaling
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Lowe 2008 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But nearly all political actors speak.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We caution that this validation does not obviate the need for unsupervised methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that any one method's performance will be context specific.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimating locations using existing data is often difficult and sometimes impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not matchedcites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Experimental — accuracy not yet measuredIn the citing paper
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
In the citing paper
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 25,941
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,346
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
(1967) · Cited by 23,719
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Items 701–800 of 3,308
Exploring the History of American Sociology Through Topic Modelling
Giordan, G., Saint‐Blancat, C., & Sbalchiero, S. (2018) · Quantitative methods in the humanities and social sciences · doi:10.1007/978-3-319-97064-6_3 · Cited by 24
Sun, M., Liu, J., Zhu, J., & LeClair, Z. (2019) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373719869318 · Cited by 24
“Such computer-assisted techniques (e.g., text reuse, topic modeling, classification methods) allow for systematic analysis of large-scale text collections without massive funding support (Grimmer & Stewart, 2013).”
BackgroundMethodThe Potential of Automated Text Analytics in Social Knowledge Building
Németh, R., & Koltai, J. (2021) · Computational social sciences · doi:10.1007/978-3-030-54936-7_3 · Cited by 30
Political frames of public health crises: Discussing the opioid epidemic in the US Congress
Weiss, M., & Zoorob, M. J. (2021) · Social Science & Medicine · doi:10.1016/j.socscimed.2021.114087 · Cited by 25
Emergent affordances and potential challenges of mobile learning apps: insights from online reviews
Gholizadeh, M., Akhlaghpour, S., Isaias, P. T., & Namvar, M. (2021) · Information Technology and People · doi:10.1108/itp-05-2021-0412 · Cited by 24
Narrative and computational text analysis in business and economic history
Ferguson‐Cradler, G. (2021) · Scandinavian Economic History Review · doi:10.1080/03585522.2021.1984299 · Cited by 25
“Thus, the refrain, as urged by Grimmer and Stewart (2013), to ‘ verify, verify, verify ’ would apply in spades here.”
BackgroundPandemic Populism? How Covid-19 Triggered Populist Facebook User Comments in Germany and Austria
Thiele, D. (2022) · Politics and Governance · doi:10.17645/pag.v10i1.4712 · Cited by 24
“Because automated content analyses raise questions of validity (Grimmer & Stewart, 2013), this study validated all measurements thoroughly, as reported in the next section.”
MethodWhen subordinates do not follow: A typology of subordinate resistance as perceived by leaders
van der Velde, A., & Gerpott, F. H. (2023) · The Leadership Quarterly · doi:10.1016/j.leaqua.2023.101687 · Cited by 29
Baturo, A., & Mikhaylov, S. (2014) · Europe Asia Studies · doi:10.1080/09668136.2014.926716 · Cited by 25
“8For a recent review see Grimmer and Stewart (2013).”
BackgroundSupervised Event Coding From Text Written in Spanish
Osorio, J., & Reyes, A. (2016) · Social Science Computer Review · doi:10.1177/0894439315625475 · Cited by 24
“Coding accuracy is a central concern in computerized textual annotation (Grimmer & Stewart, 2013; King & Lowe, 2003; Schrodt & Gerner, 1994).”
Background“Judge Lynch” in the Court of Public Opinion: Publicity and the De-legitimation of Lynching
Weaver, M. (2019) · American Political Science Review · doi:10.1017/s0003055418000886 · Cited by 25
Asking too much? The rhetorical role of questions in political discourse
Zhang, J., Spirling, A., & Danescu-Niculescu-Mizil, C. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1164 · Cited by 25
“Finally, our work contributes to a rapidly growing area of NLP applications to political domains (Monroe et al., 2008; Card et al., 2016; Gonzalez-Bailon et al., 2010; Niculae et al., 2015; Grimmer and Stewart, 2013; Grimmer et al., 2012; Iyyer et al., 2014b, inter alia).”
BackgroundEbadi, A., Tremblay, S., Goutte, C., et al. (2020) · Journal of Informetrics · doi:10.1016/j.joi.2020.101018 · Cited by 24
Muchlinski, D., Yang, X., Birch, S., Macdonald, C., & Ounis, I. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.32 · Cited by 24
Lost in Space: Geolocation in Event Data
Lee, S. J., Liu, H., & Ward, M. D. (2019) · Open Access at Essex (University of Essex) · Cited by 24
Hopkins, D. J. (2012) · Political Behavior · doi:10.1007/s11109-017-9418-4 · Cited by 35
“As with similar models in political science (Quinn et al. 2010; Roberts et al. 2014), this approach can be highly valuable in reducing the dimension of a textual data set and partitioning a set of documents into meaningful subtopics (Grimmer and Stewart 2013).5”
MethodResultBarari, S., & Simko, T. (2023) · Scientific Data · doi:10.1038/s41597-023-02044-y · Cited by 28
The Shifting Context of Sustainability: Growth and the World Ocean Regime
Jacques, P. J., & Lobo, R. (2018) · Global Environmental Politics · doi:10.1162/glep_a_00480 · Cited by 23
“Laver and Garry (2000) and Grimmer and Stewart (2013) note that QCA requires layers of validation and testing.”
BackgroundDie politische Agenda Deutschlands
Breunig, C., & Schnatterer, T. (2020) · Politische Vierteljahresschrift · doi:10.1007/s11615-020-00226-6 · Cited by 23
Consequences of Group Style for Differential Participation
Carlsen, H. B., Toubøl, J., & Ralund, S. (2020) · Social Forces · doi:10.1093/sf/soaa063 · Cited by 22
“This enables us to handle some of the very valid worries about measurement error in large-scale text analysis (Grimmer and Stewart 2013).”
BackgroundMethodReveilhac, M., & Morselli, D. (2022) · Political Research Exchange · doi:10.1080/2474736x.2022.2029217 · Cited by 30
“Grimmer and Stewart (2013) summarize di ff erent content analytical techniques lying on a continuum from dictionary-based, to (semi-)supervised (e.g. machine learning from a labelled training set), and fully data-© driven methods (e.g. topic modelling).”
BackgroundMethodResultInfluentialHeidenreich, T., Eberl, J., Lind, F., & Boomgaarden, H. G. (2022) · New Media & Society · doi:10.1177/14614448221074648 · Cited by 23
“To do so, we used an automated dictionary approach (Grimmer and Stewart, 2013) to classify each of the migration-related messages on a total of four frequently occurring issue-specific frames known from the literature on public migration discourses, namely the economic, labour market, security, and…”
MethodMeasuring digitalization capabilities using machine learning
Yang, J., Liu, J., Yao, Z., & Ma, C. (2024) · Research in International Business and Finance · doi:10.1016/j.ribaf.2024.102380 · Cited by 32
Greene, K. T., Park, B., & Colaresi, M. P. (2018) · Political Analysis · doi:10.1017/pan.2018.11 · Cited by 23
“Because we are using the underlying texts, we are also able to examine which lexical features have changed in importance over time (Monroe, Colaresi, and Quinn 2008; Quinn et al. 2010; Grimmer and Stewart 2013).”
MethodHow interagency coordination is affected by agency policy autonomy
Bjurstrøm, K. H. (2019) · Public Management Review · doi:10.1080/14719037.2019.1679236 · Cited by 23
“Automated methods should ‘[. . .] be evaluated on their ability to perform some useful scientific task’ (Grimmer and Stewart 2013, 270).”
MethodInfluentialLate Modern English Medical Texts
(2019) · John Benjamins Publishing Company eBooks · doi:10.1075/z.221 · Cited by 24
Diversionary cheap talk: economic conditions and US foreign policy rhetoric, 1945-2010
Carter, E. B. (2019) · International Interactions · doi:10.1080/03050629.2020.1688319 · Cited by 23
“19 Finally, several studies have addressed the general limitations of text as data; analyses of the Global Diplomacy Dataset should be aware of these as well (Grimmer and Stewart 2013; Schrodt 2012b; Schrodt, Van, and Brackle 2013).”
BackgroundPuschmann, C., Ausserhofer, J., & Šlerka, J. (2020) · European Journal of Communication · doi:10.1177/0267323120922068 · Cited by 22
Bérubé, M., Tang, T.-U., Fortin, F., Ozalp, S., Williams, M. L., & Burnap, P. (2020) · Forensic Science International · doi:10.1016/j.forsciint.2020.110364 · Cited by 22
“LDA topic modeling implies a certain degree of subjectivity in identification of topics, since they were not named by the computer (Grimmer and Stewart, 2013).”
BackgroundMethodInfluentialFrom Isolation to Radicalization: Anti-Muslim Hostility and Support for ISIS in the WestPreprint
Mitts, T. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2795660 · Cited by 23
Is Automated Topic Model Evaluation Broken?: The Incoherence of CoherencePreprint
Hoyle, A., Goel, P., Peskov, D., Hian-Cheong, A., Boyd-Graber, J. L., & Resnik, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02173 · Cited by 93
“Topic models are a machine learning technique widely used outside computer science, including political science (Grimmer and Stewart, 2013; Isoaho et al., 2021), social and cultural studies (Mohr and Bogdanov, 2013), digital humanities (Meeks and Weingart, 2012), and bioinformatics (Liu et al.,…”
Sajjadiani, S., Daniels, M., & Huang, H. (2022) · Personnel Psychology · doi:10.1111/peps.12538 · Cited by 22
Jackson, H. (2023) · Media War & Conflict · doi:10.1177/17506352231178148 · Cited by 23
“Experts in content analysis expound that the results of any quantitative analysis must be validated with statistics and qualitative measures since they can otherwise be misleading (Grimmer and Stewart, 2013: 4–5).”
BackgroundIntroducing the PeaceKeeping Operations Corpus (PKOC)
Amicarelli, E., & Salvatore, J. D. (2021) · Journal of Peace Research · doi:10.1177/0022343320978693 · Cited by 22
“PKOC connects the peacekeeping research community to these exciting trends in quantitative political science (Benoit et al., 2016; Grimmer & Stewart, 2013).”
BackgroundEnhancing the Detection of Criminal Organizations in Mexico using ML and NLP
Osorio, J., & Beltran, A. (2020) · IEEE International Joint Conference on Neural Network · doi:10.1109/ijcnn48605.2020.9207039 · Cited by 22
“For example, scholars use text-as-data to identify policy changes, actors, and topics relevant to voters [14].”
BackgroundCornut, J., & de Zamaróczy, N. (2020) · Cooperation and Conflict · doi:10.1177/0010836720972426 · Cited by 30
“Benefiting from the ‘big data’ explosion, equipped with newly developed computational techniques, and utilizing neopositivist methods, QTA is on the rise in political science and IR (Connelly et al., 2020; Grimmer and Stewart, 2013).”
MethodFraming the frontier – Tracing issues related to soybean expansion in transnational public spheres
Mempel, F., & Corbera, E. (2021) · Global Environmental Change · doi:10.1016/j.gloenvcha.2021.102308 · Cited by 24
All (Mayoral) Politics Is Local?
Das, S., Sinclair, B., Webster, S. W., et al. (2021) · The Journal of Politics · doi:10.1086/716945 · Cited by 22
“(4)It is now well-established that text analytics can contribute to quantifying and measuring some important characteristics of political speech, for example partisanship (Lin, Xing and Hauptmann, 2008; Grimmer and Stewart, 2013; Gentzkow and Shapiro, 2010; Ahmed and Xing, 2010; Iyyer et al., 2014).”
BackgroundAnalytical sociology amidst a computational social science revolution
Jarvis, B. F., Keuschnigg, M., & Hedström, P. (2021) · Routledge eBooks · doi:10.4324/9781003024583-4 · Cited by 24
Why It Matters What Autocrats Say: Assessing Competing Theories of Propaganda
Boussalis, C., Dukalskis, A., & Gerschewski, J. (2022) · Problems of Post-Communism · doi:10.1080/10758216.2021.2012199 · Cited by 27
“…to reduce the dimensionality of the data to increase computational efficiency while ensuring that meaningful and predictive terms are preserved (see Grimmer and Stewart 2013, 6).5 These processed texts were then used as input in the dynamic LDA algorithm to eventually produce per document dynamic…”
BackgroundMethodGregory, K., Geiger, L., & Salisbury, P. R. (2022) · Journal of Library Metadata · doi:10.1080/19386389.2022.2030635 · Cited by 22
“Volume is another area where Grimmer and Stewart observe that automated methods can be most helpful: “Automated methods…rely on having a sufficient number of words to use reliably.”
BackgroundMethodChung, G., Rodriguez, M. Y., Lanier, P., & Gibbs, D. (2022) · Journal of Technology in Human Services · doi:10.1080/15228835.2022.2036301 · Cited by 22
“Researchers must “validate, validate, validate” their results (Grimmer & Stewart, 2013, p. 5).”
BackgroundMethodInfluentialEngaged Pluralism: The Importance of Commitment
Box‐Steffensmeier, J. M. (2022) · Perspectives on Politics · doi:10.1017/s1537592721003856 · Cited by 23
Are Neural Topic Models Broken?
Hoyle, A., Goel, P., Sarkar, R., & Resnik, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.390 · Cited by 47
Revisiting Automated Topic Model Evaluation with Large Language Models
Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.581 · Cited by 52
Energy justice: Lessons from offshore wind farm siting conflicts in South Korea
Park, S., Yun, S., & Cho, K.-J. (2024) · Energy Policy · doi:10.1016/j.enpol.2023.113972 · Cited by 23
Racial Mediation in the Coverage of Candidates' Political Viability: A Comparison of Approaches
Tolley, E. (2015) · Journal of Ethnic and Migration Studies · doi:10.1080/1369183x.2014.1002201 · Cited by 22
“The goal is not to eliminate either man or machine from the process, but instead to ‘identify the best way to use both humans and automated methods for analysing texts’ (Grimmer and Stewart 2013, 270).”
BackgroundMethodInfluentialContent Analysis of Mediated Associations: An Automated Text-Analytic Approach
Arendt, F., & Karadas, N. (2017) · Communication Methods and Measures · doi:10.1080/19312458.2016.1276894 · Cited by 22
“This can be very time consuming, requires human experts, is expensive, and has to be validated (Grimmer & Stewart, 2013; Keyling, 2014; Krippendorff, 2004; Scharkow, 2012).”
BackgroundMethodInfluential75 years of dryland science: Trends and gaps in arid ecology literature
Greenville, A. C., Dickman, C. R., & Wardle, G. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0175014 · Cited by 24
Kim, S., Kwak, M., Cardozo‐Gaibisso, L., Buxton, C., & Cohen, A. S. (2017) · The Journal of Writing Analytics · doi:10.37514/jwa-j.2017.1.1.05 · Cited by 22
Using Radical Environmentalist Texts to Uncover Network Structure and Network Features
Almquist, Z. W., & Bagozzi, B. E. (2017) · Sociological Methods & Research · doi:10.1177/0049124117729696 · Cited by 22
Incivility and confrontation in online conflict discourses
Lewandowska‐Tomaszczyk, B. (2017) · Lodz Papers in Pragmatics · doi:10.1515/lpp-2017-0017 · Cited by 23
“…in the degree of emotional load and dynamics axis in Polish versus English social and political internet discourses was identified earlier in corpus linguistics (Grimmer and Stewart 2013) and corpus-based discourse analysis (Kopytowska 2013; Lewandowska-Tomaszczyk 2012, 2015, 2016a, 2016b).”
BackgroundApplied Social Science Methodology: An Introductory Guide
Gerring, J., & Christenson, D. P. (2017) · Cited by 21
Promoting alignment between innovation policy and inclusive development in South Africa
Petersen, I., & Kruss, G. (2018) · Development Southern Africa · doi:10.1080/0376835x.2018.1490175 · Cited by 22
“Like Grimmer & Stewart (2013), who point out the value of content analysis (including word frequency analysis) and the usefulness and limitations of computer-assisted analysis, we argue that our rigorous methodology provides an advance on current research practice.”
BackgroundMethodCerón, A., Curini, L., & Iacus, S. M. (2019) · SAGE Open · doi:10.1177/2158244018789229 · Cited by 22
“The supervised methods aim either to classify the individual documents into categories, via machine learning algorithms, or measuring the proportion of documents in each category, as iSA does (Grimmer & Stewart, 2013).”
BackgroundMethodInfluentialCarvalho, F., Rodrigues, R. G., Santos, G., et al. (2019) · Anais do Brazilian Workshop on Social Network Analysis and Mining (BraSNAM) · doi:10.5753/brasnam.2019.6545 · Cited by 22
“Although these processes can be quite complex, the inherent complexity of natural language texts can be reduced utilizing lexical approaches [Grimmer and Stewart 2013].”
BackgroundAnglin, K. (2019) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2019.1654576 · Cited by 22
“Dictionary-based methods require prior information of the function mapping features and classes and is therefore most appropriate when prior information on the classes is strong and where information in the text is comparatively weak (Grimmer & Stewart, 2013).”
BackgroundMethodKorsgaard, D., Bjørner, T., Sørensen, P. K., & Burelli, P. (2020) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09252-5 · Cited by 22
Do Not Blame the Media! The Role of Politicians and Parties in Fragmenting Online Political Debate
Heiberger, R. H., Majó-Vázquez, S., Castro, L., Nielsen, R. K., & Esser, F. (2021) · The International Journal of Press/Politics · doi:10.1177/19401612211015122 · Cited by 21
“928 The International Journal of Press/Politics 27(4) for a given corpus (Grimmer and Stewart 2013).”
BackgroundLi, X., Yang, A., & Yan, H. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.647670 · Cited by 23
“Bibliometric analysis is essentially an automated analysis of political texts (7).”
BackgroundMethodDokshin, F. A. (2021) · Nature Energy · doi:10.1038/s41560-021-00886-7 · Cited by 22
Who Is Doing Computational Social Science? Trends in Big Data Research A SAGE White Paper
Metzler, K., Kim, D. A., Allum, N., & Denman, A. (2016) · doi:10.4135/wp160926 · Cited by 24
McLevey, J., Crick, T., Browne, P., & Durant, D. (2022) · Canadian Review of Sociology/Revue canadienne de sociologie · doi:10.1111/cars.12378 · Cited by 21
More Laws, More Growth? Evidence from US States
Ash, E., Morelli, M., & Vannoni, M. (2025) · Journal of Political Economy · doi:10.1086/734874 · Cited by 39
Are press depictions of Affordable Care Act beneficiaries favorable to policy durability?
Chattopadhyay, J. (2015) · Politics and the Life Sciences · doi:10.1017/pls.2015.16 · Cited by 21
Iudici, A., & Corsi, A. G. (2016) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2016.11.016 · Cited by 21
Ørmen, J. (2018) · The International Journal of Press/Politics · doi:10.1177/1940161218809160 · Cited by 26
Constitutional Dialects: The Language of Transnational Legal Orders
Law, D. S. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108561792.005 · Cited by 19
Chen, T. C.-C., & Hsu, C.-H. (2020) · The British Journal of Politics and International Relations · doi:10.1177/1369148120957611 · Cited by 21
“…word segmentation on CSHRS documents, then perform unsupervised topic modelling using the stm package in R (Catalinac, 2016; Grimmer and King, 2013; Grimmer and Stewart, 2013; Jaros and Pan, 2018; King et al., 2013; Lowe and Benoit, 2013; Lucas et al., 2015; Monroe et al., 2008; Pan, 2019; Roberts…”
MethodThe Ties That Bind: Text Similarities and Conditional Diffusion among Parties
Düpont, N., & Rachuj, M. (2021) · British Journal of Political Science · doi:10.1017/s0007123420000617 · Cited by 20
“We then apply common procedures to construct a DFM (Reber 2018): we first split n-grams into unigrams that may be present from translating compound words because ‘n-grams do little to enhance performance’ (Grimmer and Stewart 2013, 272).”
BackgroundMethodIvonchyk, M., & Moldogaziev, T. T. (2021) · Urban Geography · doi:10.1080/02723638.2021.1893492 · Cited by 21
“Several approaches are available for a computational text analysis and the choice largely depends on researchers’ objectives (Grimmer & Stewart, 2013).”
BackgroundCanziani, B. F., & Nemati, H. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104377 · Cited by 21
Illiberal Communication and Election Intervention during the Refugee Crisis in Germany
Elshehawy, A., Gavras, K. L., Marinov, N., Nanni, F., & Schoen, H. (2021) · Perspectives on Politics · doi:10.1017/s1537592721003108 · Cited by 21
Measuring the diffusion of conspiracy theories in digital information ecologies
Heft, A., & Buehling, K. (2022) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565221091809 · Cited by 22
“Topic models can be applied to case-speci fi c, pre-classi fi ed corpora (such as those speci fi ed by keyword approaches) or non-case-speci fi c text corpora (for a general review of the concepts and applications of automated content analysis see Grimmer and Stewart, 2013).”
Bleich, E., Callison, J. P., Edwards, G. G., Fichman, M., Hoynes, E., Jabari, R., & van der Veen, A. M. (2018) · Journalism · doi:10.1177/1464884918818252 · Cited by 21
“2 To analyze the tone of coverage, we use automated sentiment analysis techniques similar to those applied by social scientists to study a variety of topics (Grimmer and Stewart, 2013; Hopkins and King, 2010; Young and Soroka, 2012).”
MethodGruber, J. B. (2022) · The International Journal of Press/Politics · doi:10.1177/19401612221102058 · Cited by 22
“…to significantly reduce the resources needed to code texts by employing machine learning and other computational techniques for content analysis have become more advanced and are more regularly employed in the social sciences in the last decade (see Grimmer and Stewart, 2013, for an overview).”
BackgroundMethodInfluentialPark, H.-J., & Grundmann, P. (2022) · Journal of Environmental Policy & Planning · doi:10.1080/1523908x.2022.2094353 · Cited by 27
“When it comes to the number of topics, we gave priority to the ‘substantive fit’ of the model that discloses substantive information and helps answer our research question (Grimmer & Stewart, 2013; Jacobs & Tschötschel, 2019) to the extent that the statistical fit is not significantly compromised.”
BackgroundHur, D.-K., Lee, S., & Kim, H. (2022) · Tourism Management Perspectives · doi:10.1016/j.tmp.2022.100991 · Cited by 21
Does Violence Against Journalists Deter Detailed Reporting? Evidence From Mexico
Dorff, C., Henry, C., & Ley, S. (2022) · Journal of Conflict Resolution · doi:10.1177/00220027221128307 · Cited by 24
“Processing text from PDFs with optical character recognition or other techniques with an acceptable level of accuracy is often expensive (Grimmer and Stewart 2013).”
BackgroundKaran, M., Šnajder, J., Širinić, D., & Glavaš, G. (2016) · SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities · doi:10.18653/v1/w16-2102 · Cited by 21
“Understanding politics means understanding what political actors are saying and writing (Grimmer and Stewart, 2013), i.e., understanding the content of the messages.”
BackgroundStable Topic Modeling with Local Density Regularization
Koltcov, S., Nikolenko, S., Koltsova, O. Y., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45982-0_16 · Cited by 21
“Apart from exploratory analysis of large text corpora, topic modeling can also be used to mine latent variables from the documents such as [12,18].”
MethodGrill, C., & Boomgaarden, H. G. (2017) · European Journal of Communication · doi:10.1177/0267323117725971 · Cited by 21
“Since the meaning of a text can be derived from the words that the text is composed of regardless of the specific word order within that text, this study relies on the ‘bag of words’ approach (Grimmer and Stewart, 2013).”
MethodUsing machine learning to help vulnerable tenants in New York city
Ye, T., Johnson, R., Fu, S., Copeny, J., Donnelly, B., Freeman, A., Lima, M., Walsh, J., & Ghani, R. (2019) · ACM SIGCAS Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3314344.3332484 · Cited by 22
“For example, it has been introduced to forecast issues of criminal justice [4], detect online rumors on social media [25], identify political bias in text [11], map wealth and poverty in given areas [5, 10] and even facilitate medical diagnoses [14].”
BackgroundAssessing the impact of infrastructure projects on global sustainable development goals
Mansell, P., Philbin, S. P., Broyd, T., et al. (2019) · Proceedings of the Institution of Civil Engineers - Engineering Sustainability · doi:10.1680/jensu.19.00044 · Cited by 21
Hopp, F. R., Fisher, J. T., & Weber, R. (2020) · Journal of Communication · doi:10.1093/joc/jqaa015 · Cited by 21
Are rankings and accreditation related? Examining the dynamics of higher education in Poland
Rybinski, K. I. (2020) · Quality Assurance in Education · doi:10.1108/qae-03-2020-0032 · Cited by 22
Ma, J., Ebeid, I. A., de Wit, A., Meiying, X., Yang, Y., Bekkers, R., & Wiepking, P. (2021) · VOLUNTAS International Journal of Voluntary and Nonprofit Organizations · doi:10.1007/s11266-021-00414-x · Cited by 19
Communications-based early detection of gambling-related problems in online gambling
Häfeli, J., Lischer, S., & Haeusler, J. (2014) · International Gambling Studies · doi:10.1080/14459795.2014.980297 · Cited by 21
“An overview to the practical application of automated text analysis in social sciences can be found in Grimmer and Stewart (2013).”
MethodIs the NPT unraveling? Evidence from text analysis of review conference statements
Barnum, M., & Lo, J. (2020) · Journal of Peace Research · doi:10.1177/0022343320960523 · Cited by 22
“Following Grimmer & Stewart (2013), we select the number of topics for our model based on substantive fit.”
Market Reaction to the Positiveness of Annual Report NarrativesPreprint
Yekini, L. S., Wisniewski, T. P., & Millo, Y. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2544266 · Cited by 21
“Examples of papers that operationalized measures of positiveness/optimism constructed in that manner include Feldman, Govindaraj, Livnat, and Segal (2008), Kothari, Li, and Short (2009), Henry (2006), Cicon, Clarke, Ferris, and Jayaraman (2014), Wisniewski and Moro (2014), Ferguson, Philip, Lam, & Guo (2015). Cho, Roe…”
More Than Words: A Textual Analysis of Monetary Policy CommunicationPreprint
Luangaram, P., & Wongwachara, W. (2017) · RePEc: Research Papers in Economics · Cited by 21
Framing as an Information Control Strategy in Times of Crisis
Xia, S., Huang, H., & Zhang, D. (2022) · Journal of East Asian Studies · doi:10.1017/jea.2022.5 · Cited by 22
“Before using STM to estimate the topics of specific text corpora, researchers need to explicitly opt for the number of topics, yet there is no right answer to the appropriate number of topics for a given text corpora (Grimmer and Stewart 2013).”
BackgroundMethodMetaphor-Enabled Marketplace Sentiment Analysis
Luri, I., Schau, H. J., & Ghosh, B. P. (2023) · Journal of Marketing Research · doi:10.1177/00222437231191526 · Cited by 21
“…a compromise option, as automated text analysis and qualitative text analysis are not mutually exclusive, but instead can be seen as extreme points of a continuum on which computers and automated methods augment, rather than substitute, researchers ’ qualitative insight (Grimmer and Stewart 2013).”
BackgroundMethodInfluentialSebők, M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024) · Social Science Computer Review · doi:10.1177/08944393241259434 · Cited by 29
“In light of promising fi rst results and a test of external validity on Manifesto data, we argue (echoing Grimmer and Stewart (2013, p. 281)) that the proposed design can – over time – replace double-blind human coding for a multitude of comparative classi fi cation problems as the new gold standard.”
POPmine: Tracking Political Opinion on the Web
Saleiro, P., Amir, S., Silva, M. J., & Soares, C. (2015) · doi:10.1109/cit/iucc/dasc/picom.2015.228 · Cited by 20
“The increase in the digital sources of text, on the one hand, and current advances in computation power and design, on the other, are making this development both necessary and possible, while also raising awareness about the inferential pitfalls involved [7, 8].”
BackgroundThe Perils of Classifying Political Orientation From Text.
Hao, Y., Lavoie, A., & Das, S. (2017) · International Joint Conference on Artificial Intelligence · Cited by 21
Antons, D., Joshi, A. M., & Salge, T. O. (2018) · RWTH Publications (RWTH Aachen) · doi:10.25384/sage.c.4123424 · Cited by 20
Electoral Punishment and Protest Politics in Times of Crisis
Bremer, B., Hutter, S., & Kriesi, H. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108891660.011 · Cited by 20
Statistical Modeling: The Three Cultures
Daoud, A., & Dubhashi, D. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.89f6fe66 · Cited by 31
“For example, mobilizing research assistants to code up the political content of arhival policy documents an analog method (Daoud, Reinsberg, et al. 2019); training natural language processing (NLP) algorithms to do the same thing is a digital method (Grimmer and Stewart 2013).”
Method
Metadata
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
