The DOI matches its Crossref record; 105 of the 109 references it cites were found in a registry or an index; 12 of the 89 sources its claims cite were checked for support.
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
161 Own interpretation · 38 No basis given · 28 Sources not found · 26 No full text of the sources · 21 Own result · 11 Supported · 8 Not checked yet · 1 Partly supported · 1 Own method
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Supportedcites Hand 2006 source
Section 3, “Design Sample Selection,” says the classical supervised-classification paradigm assumes the design set is randomly drawn from the same distribution as future points to be classified. That supports the citing sentence’s application of the assumption to documents; the supplied copy does not provide a page number. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract says that determining in advance which clustering objective function will produce a useful or insightful partition for a given dataset is unknown and difficult, if not logically impossible. It also says there is no method for choosing beforehand which unsupervised approach will yield the most useful clustering for a particular application. (LLM, conf=0.92)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
In Model Output (page unavailable), the paper calls its speech-topic assignments “parameters of direct interest” and uses them to measure attention to political topics over time and by speaker group. Model Specification and Sensitivity Analysis (page unavailable) says the authors selected topic categories to correspond to areas of governmental competence. Together, these support the paraphrase. (LLM, conf=0.87)
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Supportedcites Quinn et al. 2010 source
The model description explains that speech topics are inferred and their probabilities vary by day. The Abstract reports a daily series of attention to topics in U.S. Senate speeches. These passages support the quoted claim; they do not establish the separate 'first topic model' characterization in the citing sentence. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology (§2.4) says the authors project a metric space of clusterings into two Euclidean dimensions and that nearby points represent similar clusterings. Methodology (§2.3) defines similarity by how documents are grouped across clusterings, matching the citing sentence’s paraphrase. (LLM, conf=0.94)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Categorizing Texts: Methods, Assumptions, and Costs” (page not available), the paper says its topic model infers categories rather than requiring them to be fixed beforehand, and shifts effort to postanalysis. “Topic Interpretation and Intratopic Semantic Validity” (page not available) describes examining the resulting topics and documents to assess their validity. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“empirically these sentences are rare.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the restrained category, the recall is 0.75.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 158
- It may state two separate claims as one. 34
Every source the paper cites for this claim was checked and supports it.
In the paper
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Supportedcites Berinsky, Huber, and Lenz 2012 source
The Introduction says MTurk can facilitate low-cost experiments, and Section 2 describes how researchers post tasks for workers and recruit survey participants. The Abstract also describes MTurk experiments as easy to field. Together, these passages support the claim that the platform can make data acquisition efficient. (LLM, conf=0.91)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Supportedcites Monroe, Colaresi, and Quinn (2008) source
Section 3 describes methods for selecting and evaluating words that distinguish Democratic from Republican Senate speeches. Section 3.1 makes clear that the categories are already identified: the speakers’ partisanship is observed, not something the methods need to infer. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Grimmer and King 2011 source
Section 3.1 (page unavailable) defines cluster quality from human judgments as within-cluster similarity minus between-cluster similarity, and Section 3 reports using that measure to compare clusterings. The measure is comparative rather than an absolute test of semantic validity, consistent with the paper’s caution that there is no reliable gold-standard clustering for this purpose. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Abstract (p. 2643) says the authors develop a space containing existing clustering methods and millions of additional solutions based on combinations of existing clusterings, then enable users to explore it quickly. Section 2, Methodology, describes how users navigate that space to select useful clusterings. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
What the sources say
- Supportedcites Grimmer and King 2011 source
The Methodology section describes CAC as organizing a large number of possible clusterings in a visual space so researchers can efficiently select useful ones. The Abstract likewise says users can explore a space containing existing methods and millions of additional solutions and quickly identify useful clusterings. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
What the sources say
- Supportedcites Quinn et al. 2010 source
In “Exogenous Events and Predictive Validity,” the paper defines predictive validity as correspondence between a measure and external events, then links spikes in topic attention to events including 9/11 and debates over military action. The Discussion says examining relationships with such events helps assess the validity of its attention measures. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The topics in Table 4 exemplify the utility of unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Semantic Validation via Experiments:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Language is the medium for politics and political conflict.”
- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Politics and political conflict often occur in the written and spoken word.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential when applying automated content methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it substantially reduces the costs of analyzing large collections of text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“to understand what politics is about we need to know what political actors are saying and writing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“require extensive and problem-specific validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they are no substitute for careful thought and close reading”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance of any one classifier can vary substantially across contexts”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“And hiring coders to manually read all documents is still very expensive.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One way to improve upon dictionaries are supervised methods”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“automated content methods are incorrect models of language.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finding the separating words is also relatively easy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validation is essential.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tone is just one type of analysis a dictionary method can perform.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is often difficult, and sometimes impossible, to validate the output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe described above is one way to represent text as data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather than replace humans, computers amplify human abilities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is also substantial variation within families of models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“validations are necessary to establish its meaning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- The authors state this as an assumption: Ideology dominates the language used in the text.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The data generation process for any text (including this one) is a mystery-even to linguists.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of applied research uses stemming.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“sometimes the problem calls for a completely different approach.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stemmer is a much cruder algorithm, but considerably faster.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance does not seem to matter for most applications”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“automated scraping methods make acquiring these data easier (Jackman 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not matchedcites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These programs simply provide the researcher with output.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying FAC algorithms to texts may have mixed results”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 4.1 Alternative Methods for Representing Text
What the sources say
- Not checked — no full text of the sourcecites Spirling (2012) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.3 Principle 3: There Is No Globally Best Method for Automated Text Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Eggers and Hainmueller 2009 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Automated methods also rely on having a sufficient number of words to use reliably.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
What the sources say
- Not checked — no full text of the sourcecites Porter 1980 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
What the sources say
- Not matchedcites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these require the close reading of texts and thoughtful analysis by the researcher.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assigning texts to categories is the most common use of content analysis methods in political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5 Classifying Documents into Known Categories
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“not all of language's complexity is necessary to effectively analyze texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.1 Principle 1: All Quantitative Models of Language Are Wrong-But Some Are Useful
What the sources say
- Not matchedcites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.2 Principle 2: Quantitative Methods Augment Humans, Not Replace Them
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 2.4 Principle 4: Validate, Validate, Validate
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.3
What the sources say
- Not checked — no full text of the sourcecites van der Vaart, Dudoit, and van der Laan 2006 source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
- Not matchedcites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
What the sources say
- Not checked — no full text of the sourcecites Hastie, Tibshirani, and Friedman 2001 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Dictionary methods require scholars to identify words that separate classes beforehand.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2 Supervised Learning Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
What the sources say
- Not matchedcites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number necessary will depend upon the specific application of interest.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
- Not checked — no full text of the sourcecites Hillard, Purpura, and Wilkerson 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.1 Dictionary Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAC methods are useful, but are necessarily model dependent.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.1 Constructing a training set
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1 FAC
What the sources say
- Not matchedcites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- The authors state this as an assumption: Given a document's category, the words are generated independently.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Supervised and dictionary methods assume a well-defined set of categories.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.4 Applying supervised learning methods: Russian military discourse
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > Measuring Proportions:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Yiannakis 1982 source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the K-means' objective function is multimodal and non-continuous.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
In the paper
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
What the sources say
- Not checked — no full text of the sourcecites Blei 2012 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and therefore several random restarts are often necessary to find the global optimum.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Also stated in: § 6.1.2 Mixed membership models · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The goal is revelation of substantively interesting information.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“To estimate a topic, the models use the co-occurrence of words across documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Situating the models in a data-generating process makes including additional structure straightforward.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The preprocessed texts represent a substantial simplification of the documents.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.3 Setting the Number of Clusters
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“CAC methods provide a way to explore these different assumptions.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.2 Mixed membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, CAC provides an accelerated way to explore new collections of texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many other distance metrics are possible, and each will lead to different clusterings.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.1.1 Single membership models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But CAC does place substantial burden on users to identify clusterings that are useful.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.2 CAC
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4 Validating Unsupervised Methods: How Legislators Present Their Work to Constituents
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.1 Validating topics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.2 Unsupervised Methods for Scaling
What the sources say
- Not checked — no full text of the sourcecites Slapin and Proksch 2008 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Laver, Benoit, and Garry 2003 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not checked — no full text of the sourcecites Lowe 2008 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“But nearly all political actors speak.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We caution that this validation does not obviate the need for unsupervised methods.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that any one method's performance will be context specific.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Estimating locations using existing data is often difficult and sometimes impossible.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
What the sources say
- Not matchedcites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. It may be too vague to check against evidence.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7 Measuring Latent Features in Texts: Scaling Political Actors
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 6.4.2 Validating expressed priorities
What the sources say
- Not matchedcites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 7.3 Applying Unsupervised Methods to Political Texts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Experimental — accuracy not yet measuredIn the citing paper
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
In the citing paper
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
§ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts > 5.2.2 Applying a supervised learning model
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Modern Applied Statistics with S
Venables, W. N., & Ripley, B. D. (2002) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 25,941
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
(1967) · Cited by 23,719
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Items 1,901–2,000 of 3,308
Small Data Problems in Political Research: A Critical Replication Study
de Vos, H., & Verberne, S. (2022) · LDV-Forum/Journal for language technology and computational linguistics · doi:10.21248/jlcl.35.2022.226 · Cited by 2
“One of the first major papers on the use of automatic text analysis in the field was Grimmer and Stewart (2013).”
BackgroundМоногарова, А. Г., Shiryaeva, T., & Arupova, N. (2021) · Journal of language and Education · doi:10.17323/jle.2021.13371 · Cited by 4
“…the prior corpus-based research built on the premise that collecting and analyzing large amounts of discourse samples is an effective tool for understanding the way people communicate (e.g., Biber et al., 1998; Stubbs, 2001; McCarthy & Carter, 2001; Budge & Pennings, 2007; Grimmer & Stewart, 2013).”
Kelm, O., Gerl, K., & Meißner, F. (2020) · Springer eBooks · doi:10.1007/978-3-658-26242-6_55-1 · Cited by 3
Studying Religious Change: Combining Close and Distant Reading in the Field of Sociology of Religion
Lundgren, L., & Jensdotter, L. (2022) · Nordic Journal of Religion and Society · doi:10.18261/njrs.35.2.4 · Cited by 3
TipScreener: A Framework for Mining Tips for Online Review Readers
Luo, H., Song, W., & Zhou, W. (2022) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer17040087 · Cited by 3
Sebők, M., Kiss, R., & Járay, I. P. (2023) · Journal of the Knowledge Economy · doi:10.1007/s13132-023-01395-6 · Cited by 3
“The widespread use of complex computer analysis of large text corpora and statistical techniques started in the 2000s (Boumans & Trilling, 2016; Grimmer & Stewart, 2013).”
Decoding data analytics capabilities from topic modeling on press releases
Bonilla, J., & Rao, B. (2015) · Portland International Conference on Management of Engineering and Technology (PICMET) · doi:10.1109/picmet.2015.7273249 · Cited by 3
“Some of these models models have been used by Blei [6] on academic papers, by Newman and Block [5] on newspapers, by Quinn et al on Senate floor speeches [12], and by Hong and Davis [15] on Twitter data [26].”
BackgroundMethodInfluentialUsing the web to support political analysis
Loyola, P., Szederkenyi, F., & Matsuo, Y. (2016) · ACM Web Science Conference (WebSci) · doi:10.1145/2908131.2908166 · Cited by 3
“On the other hand, Grimmer et al [12] performs a deep review of the state of the art, providing a taxonomy of methods and tools.”
BackgroundUsing Vector Space Models to Understand the Circulation of Habeas Corpus in Hawai'i, 1852–92
Romney, C. W. (2016) · Law and History Review · doi:10.1017/s0738248016000353 · Cited by 3
The diffusion of policy frames: evidence from a structural topic model
Gilardi, F., Shipan, C. R., & Wüest, B. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-143864 · Cited by 3
Fabo, P. R., Plancq, C., & Poibeau, T. (2016) · univOAK (4 institutions : Université de Strasbourg, Université de Haute Alsace, INSA Strasbourg, Bibliothèque Nationale et Universitaire de Strasbourg) · Cited by 3
Introduction: Analysing Organized Interests and Public Opinion Towards Welfare Reforms
Ebbinghaus, B., & Naumann, E. (2017) · Springer eBooks · doi:10.1007/978-3-319-63652-8_1 · Cited by 3
Transitory and Resilient Salient Issues in Party Manifestos, Finland, 1880s to 2010s
Ahonen, P., & Koljonen, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01129-1_2 · Cited by 3
“Following good practices of computational examination, ‘stop‐ words’ were eliminated from the textual data [13, 19, 23, 37].”
MethodSearching for a Reference: Using Automated Text Analysis to Study Judicial Compliance
Šipulová, K., Smekal, H., & Janovský, J. (2018) · Masaryk University Journal of Law and Technology · doi:10.5817/mujlt2018-2-2 · Cited by 3
“Most of the existing research focuses on the relationship between domestic courts and the Court of Justice of the European Union (“CJEU”) or the EctHR.(11) Our research focuses on the use of ECtHR case law by the Czech apex courts – the Constitutional Court, the Supreme Court and the Supreme Administrative Court.”
BackgroundBioética y catolicismo en la Argentina contemporánea: Un análisis en base a seis campos en disputa
Prieto, S. (2018) · Religião & Sociedade · doi:10.1590/0100-85872018v38n2cap01 · Cited by 3
The diffusion of policy perceptions: evidence from a structural topic model
Gilardi, F., Shipan, C. R., & Wüest, B. (2015) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-143867 · Cited by 3
Rozgonyi, K. (2019) · Journal of Digital Media & Policy · doi:10.1386/jdmp.10.1.67_1 · Cited by 3
The Congressional Classification Challenge
Yan, H., Das, S., Lavoie, A., Li, S., & Sinclair, B. (2019) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3328526.3329582 · Cited by 3
“Nevertheless, it is recognized that “dictionary based” methods for classifying political text have trouble generalizing across dierent domains of text [27].”
BackgroundWhat Data Gathering Strategies Should I Use?
Cooksey, R., & Mcdonald, G. M. (2019) · Springer eBooks · doi:10.1007/978-981-13-7747-1_14 · Cited by 3
Fink, S., Hagendorf, W., & Klinger, N. (2019) · dms – der moderne staat – Zeitschrift für Public Policy Recht und Management · doi:10.3224/dms.v12i1.05 · Cited by 3
Atzpodien, D. S. (2020) · Zeitschrift für Vergleichende Politikwissenschaft · doi:10.1007/s12286-020-00445-6 · Cited by 3
Writing Gender Out or Working It Back In? Media Coverage of Child Benefits in Canada
Wallace, R., & Goodyear‐Grant, E. (2020) · Journal of Women Politics & Policy · doi:10.1080/1554477x.2020.1783932 · Cited by 3
Mapping the German Tech Blog Sphere and Its Influence on Digital Policy
Barbaresi, A., & Pohlmann, J. (2020) · Ars digitalis · doi:10.1007/978-3-658-30702-8_7 · Cited by 3
Kovačević, A., Grljević, O., Bošnjak, Z., et al. (2020) · Poznań Studies in Contemporary Linguistics · doi:10.1515/psicl-2020-0006 · Cited by 3
Between populism and democracy: ‘the People’ in election discourse
Lavi, L. (2021) · European Political Science · doi:10.1057/s41304-021-00347-5 · Cited by 3
Voting and Social Media-Based Political Participation
Göbel, S. (2021) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2021.021 · Cited by 3
How to make project plans trustworthy: an investigation based on multi-criteria contractor selection
Ying, H., Zhang, S., Zhao, X., & Zhang, M. (2021) · International Journal of Logistics Research and Applications · doi:10.1080/13675567.2021.1961700 · Cited by 3
“We chose this method because textual content can be appropriately represented by a list of the words it contains (Grimmer and Stewart 2013; Sebastiani 2002).”
BackgroundMethodA bird’s-eye view on South Asian languages through LSI
Borin, L., Saxena, A., Comrie, B., & Virk, S. M. (2020) · Journal of South Asian Languages and Linguistics · doi:10.1515/jsall-2021-2034 · Cited by 4
“Because such methods are “ necessarily incorrect models of language, the output always necessitates careful validation ” (Grimmer and Stewart 2013: 294; original emphasis).”
MethodUnequal values: equality and race in state of the union addresses, 1960–2018
Hero, R. E., & Levy, M. E. (2021) · The Journal of Race Ethnicity and Politics · doi:10.1017/rep.2021.21 · Cited by 3
Parliaments in the Low Countries: Representing Divided Societies
de Vet, B., & Louwerse, T. (2021) · Politics of the Low Countries · doi:10.5553/plc/.000025 · Cited by 3
Boudet, H. S. (2021) · Nature Energy · doi:10.1038/s41560-021-00892-9 · Cited by 3
Using Digital Content Analysis for Online Research: Online News Media Depictions of Older Adults
Amundsen, D. (2021) · SAGE Publications Ltd eBooks · doi:10.4135/9781529798364 · Cited by 3
“Complexity of language makes it challenging for automated digital content analysis to replace careful, close reading of texts (Grimmer & Stewart, 2013), therefore validation is essential when applying automated analysis.”
MethodEvans, J. A. (2022) · Sociological Methods & Research · doi:10.1177/00491241221123086 · Cited by 4
“The book comes from three top research performers who have individually and together published leading work with these methods on topics that range across several areas of American and international politics (Feder et al. 2021; Grimmer 2009; Grimmer and Stewart 2013; King, Pan, and Roberts 2013).”
BackgroundJungherr, A. M., & Posegga, O. (2023) · Springer eBooks · doi:10.1007/978-3-658-31480-4_54-1 · Cited by 3
On the Dynamics of Narratives of Crisis during Terror Attacks
Grobelscheg, L., Sliwa, K., Kušen, E., & Strembeck, M. (2022) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams58071.2022.10062580 · Cited by 3
“Nevertheless, numerous alternative methods exist which might affect the distribution of narratives that can be found in the data-sets (see [37], [38], [39], see also Section III).”
BackgroundFeedback to align teacher and student in a Digital Learning Ecosystem
Pentucci, M., Sarra, A., & Laici, C. (2023) · EDUCATION SCIENCES AND SOCIETY · doi:10.3280/ess1-2023oa15761 · Cited by 4
“In this respect, Grimmer and Stewart (2013) observe that there is no correct number of topics that is appropriate for any given corpus, while Roberts et al .”
BackgroundFeinstein, B. D., & Nou, J. (2023) · Journal of Empirical Legal Studies · doi:10.1111/jels.12369 · Cited by 2
Between continuity and change: The EU's mechanism of differentiated value integration
Oshri, O., & Shenhav, S. R. (2017) · European Journal of Political Research · doi:10.1111/1475-6765.12225 · Cited by 3
Measuring discourse by algorithm
Caspi, A., & Stiglitz, E. H. (2019) · International Review of Law and Economics · doi:10.1016/j.irle.2019.105863 · Cited by 5
Improving assessment capability for accounts receivable pool financiers
Song, X., Ying, H., Zhao, X., et al. (2019) · Industrial Management & Data Systems · doi:10.1108/imds-04-2019-0239 · Cited by 3
Who Blames or Endorses Whom? Entity-to-Entity Directed Sentiment Extraction in News Text
Park, K., Pan, Z., & Joo, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.358 · Cited by 3
“We believe the proposed method can therefore further the current interdisciplinary efforts of the NLP, machine learning, and the social science communities (Grimmer and Stewart, 2013; Roberts et al., 2014; Joo and Steinert-Threlkeld, 2018).”
MethodDie Entwicklung des Klimawandels als politisches Problem
Ruß, D. (2014) · dms – der moderne staat – Zeitschrift für Public Policy Recht und Management · doi:10.3224/dms.v7i2.17321 · Cited by 3
Creating and Destroying Party Brands
Grimmer, J. (2014) · doi:10.3115/v1/w14-2502 · Cited by 3
Self-citationNational or European Politicians? Gauging MEPs Polarity When Russia is ConcernedPreprint
Андреевна, Д. А., Алексеевна, Х. А., & Skougarevskiy, D. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2779532 · Cited by 3
A Crowd Content Analysis Assembly Line: Annotating Text Units of AnalysisPreprint
Adams, N. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2808731 · Cited by 3
Social Media Strategies of Right-Wing Movements - The Radicalization of PegidaPreprint
Schwemmer, C. (2018) · SocArXiv (OSF Preprints) · doi:10.17605/osf.io/js73z · Cited by 3
KnowBias: A Novel AI Method to Detect Polarity in Online ContentPreprint
Saligrama, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00724 · Cited by 3
Ardağ, M. M., Cohrs, J. C., & Selck, T. J. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3321204 · Cited by 4
Using legislative speech to unveil conflict between coalition partiesPreprint
Bergmann, H., Geese, L., Koß, C., & Schwemmer, C. (2018) · doi:10.31235/osf.io/pgnwa · Cited by 3
Raveau, M. P., Couyoumdjian, J. R., Fuentes-Bravo, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.00140 · Cited by 3
“7 Text analyzing The analysis of large text corpora has proven fruitful in a range of topics in social sciences, including the analysis of discourse surrounding social movements [56], content analysis for political texts [57, 58], mapping social conflicts from legal sentences [39], research choice and the production o…”
BackgroundExploring NMF and LDA Topic Models of Swedish News Articles
Svensson, K., & Blad, J. (2020) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 3
Neural Transfer Learning with Transformers for Social Science Text Analysis.Preprint
Wankmüller, S. (2021) · arXiv (Cornell University) · Cited by 3
Gender and Lawmaking in Times of Quotas. Evidence from the French ParliamentPreprint
Lippmann, Q. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3330617 · Cited by 3
“Finally, methodologically, this paper contributes to the growing literature using text analysis methods (see Grimmer & Steward 2013 or Gentzkow et al. 2017).”
Politician-Citizen Interactions and Dynamic Representation: Evidence from TwitterPreprint
Schöll, N., Gallego, A., & Mens, G. L. (2021) · RePEc: Research Papers in Economics · Cited by 3
Masciandaro, D., Ferrara, F. M., Moschella, M., et al. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3820745 · Cited by 3
Panem Et Circenses: The Persuasive Effect of Soft News in Berlusconi's ItalyPreprint
Angelis, A. D., & Vecchiato, A. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3245572 · Cited by 3
Citizen complaints as an accountability mechanism: Uncovering patterns using topic modelingPreprint
Olivos, F., Saavedra, P., & Dammert, L. (2022) · doi:10.31235/osf.io/m6zke · Cited by 3
Abstract(s) at the core: a case study of disciplinary identity in the field of linguistics
LiCausi, T. J., & McFarland, D. A. (2022) · Higher Education · doi:10.1007/s10734-021-00810-8 · Cited by 3
Fernández-Vázquez, P. (2022) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/k9jh2 · Cited by 3
Integrating Human and Machine Coding to Measure Political Issues in Ethnic Newspaper ArticlesPreprint
Kim, J. Y. (2020) · doi:10.31235/osf.io/pg3aq · Cited by 3
Network analysis and comparative migration law: examples from the European Court of Human RightsPreprint
Byrne, W. H., Gammeltoft‐Hansen, T., & Olsen, H. P. (2023) · doi:10.31235/osf.io/9y632 · Cited by 3
Using OpenAI models as a new tool for text analysis in political leaders’ unstructured discoursePreprint
Bianchini, G. E., Zanotti, L., & Meléndez, C. (2023) · doi:10.31234/osf.io/kdngb · Cited by 3
Williams, N. W., Casas, A., Aslett, K., & Wilkerson, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4540742 · Cited by 3
Flannery, H., & Mittendorf, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4565216 · Cited by 3
The Use of LLMs to Annotate Data in Management Research: Foundational Guidelines and WarningsPreprint
Carlson, N. A., & Burbano, V. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4836620 · Cited by 4
Natural Language Processing for Innovating Behavioral Political Science Research
Li, Q. (2020) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190634131.013.33 · Cited by 2
Varieties of inclusionary and exclusionary populism in the Italian regional elections 2018-20
Paxton, F. (2021) · Contemporary Italian Politics · doi:10.1080/23248823.2021.1969785 · Cited by 2
“Such an analysis represents text as data and performs word counts to measure the extent to which documents belong to a particular category (Grimmer and Stewart 2013).”
Policymaking in personalist dictatorships: a theory of outbidding
White, A. C. (2023) · Policy Studies · doi:10.1080/01442872.2023.2200244 · Cited by 2
“Following the existing literature (Grimmer and Stewart 2013), preprocessing actions were performed to create stem word tokens, removing punctuation, numbers, stopwords and symbols.”
MethodLost in communication: The vanished momentum of just transition in South Korea
Lee, H., Kang, M., & Lee, E. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103642 · Cited by 2
Does CEO guilt influence the adoption of employee welfare practices?
Ashfaq, S., Riaz, H., Mujtaba, G., Saeed, A., & Baloch, M. S. (2024) · European Management Review · doi:10.1111/emre.12675 · Cited by 2
Mind the context! The role of theoretical concepts for analyzing legislative text data
Schwalbach, J. (2024) · Research & Politics · doi:10.1177/20531680241277466 · Cited by 2
“Fifth, from a measurement perspective, it is important to keep in mind that the “ complexity of language implies that all methods necessarily fail to provide an accurate account of the data generating process used to produce texts ” (Grimmer and Stewart, 2013).”
Shaping the Narrative: Examining News Coverage of Voter ID Laws in the United States
Alonso-Curbelo, A., Trilling, D., Simon, M., & Kroon, A. (2024) · Journalism Studies · doi:10.1080/1461670x.2024.2412211 · Cited by 1
“…appropriate because it is particularly well-suited to analyse emerging phenomena (Walter and Ophir 2019), and allows researchers to identify patterns and relationships in texts that may not be immediately apparent through traditional methods of language analysis (Grimmer and Stewart 2013).”
McGowan, S. K., Corrales-Martinez, M.-J., Brender, T. D., Smith, A. K., Kim, S., Harrison, K. L., Mills, H., Lee, A., Bamman, D., & Cobert, J. M. (2024) · Medical Decision Making · doi:10.1177/0272989x241293446 · Cited by 2
“28 Leveraging domain experts to inform our understanding of clinical uncertainty could improve the validity of existing NLP tools and provide a foundation for accurately measuring uncertainty within notes.”
Psychological reactance to vaccine mandates on Twitter: a study of sentiments in the United States
Hsieh, P.-H. (2025) · Journal of Public Health Policy · doi:10.1057/s41271-025-00554-0 · Cited by 2
“Supervised learning methods were employed for text classification to identify tweets related to vaccine mandates [21].”
Popa, M. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00353-9 · Cited by 2
Congressional rhetoric on Twitter and the crisis of democracy
Miller, C. J., & Jiang, Y. (2025) · Communication and Democracy · doi:10.1080/27671127.2025.2478863 · Cited by 3
Loewe, M., & Ott, I. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.102897 · Cited by 2
Media, Digital Engagement, and the Dynamics of Attitude Change Toward Foreigners in Switzerland
Reveilhac, M. (2025) · Journal of International Migration and Integration / Revue de l integration et de la migration internationale · doi:10.1007/s12134-025-01264-y · Cited by 5
Aligning Emotions: A Comparative Analysis of Text and Imagery by European Party Leaders on Instagram
Decadri, S., Mancosu, M., Negri, F., & Scaduto, G. (2025) · The International Journal of Press/Politics · doi:10.1177/19401612251339107 · Cited by 3
“We validated the TRBSLs classification following gold-standard research practices (Grimmer and Stewart 2013).”
Global cultural appropriation discourse on reddit: a topic modeling approach
Mostafa, M. M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11277-4 · Cited by 3
“However, since there is no agreement in the literature on what constitutes the ‘‘right’’ number of topics, this depends on the utility and interpretability of the research question [89].”
Özvatan, Ö. (2025) · Springer eBooks · doi:10.1007/978-3-658-48202-2_10 · Cited by 2
China’s Smart Energy Policy Evaluation Based on Policy Modelling Consistency Index
Cai, R., Zhang, T., Wang, X., Zhao, S., Yang, H., & Geng, Q. (2025) · Energies · doi:10.3390/en18205339 · Cited by 2
Dirty talk: Media discourse and the struggle over South Africa’s coal transition
Bez, C. S., Raederscheidt, G., & Steckel, J. C. (2025) · Energy Research & Social Science · doi:10.1016/j.erss.2025.104398 · Cited by 2
“Recognizing the limitations of traditional discourse analysis tools for dealing with large text corpora [12,27], recent literature increasingly employs statistical techniques that treat text as data.”
Lin, H.-C. K., Lee, Y.-J., Tsao, H., & Lin, M.-Z. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129985 · Cited by 2
Sebők, M., Buzogány, Á., Fleischer, J., Gessler, T., Takács, A., Theriault, S. M., & Holányi, Á. (2025) · Journal of European Public Policy · doi:10.1080/13501763.2025.2583176 · Cited by 4
“While double-blind human coding can still be considered to be the gold standard for many content analysis tasks, machine learning and the text-as-data approach have gained widespread popularity (Grimmer & Stewart, 2013).”
Hu, H. (2025) · Comparative European Politics · doi:10.1057/s41295-025-00445-4 · Cited by 2
“…six-dimensional framework of connectivity proposed by Gaens et al. (2023), infrastructure, economic, institutional, knowledge, societal, and security, which provides a structured lens for identifying discursive emphases and communication patterns within each initiative (Grimmer and Stewart 2013).”
Public sector innovation and wicked problems in Asia: From reactive to proactive governance
Raghavan, A., Orazgaliyev, S., & Demircioğlu, M. A. (2025) · International Journal of Innovation Studies · doi:10.1016/j.ijis.2025.12.003 · Cited by 2
Configurational impacts of government policies on electric vehicle diffusion: A global analysis
Yang, C., Qian, L., & Liu, M. (2025) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2025.105144 · Cited by 4
Sumayah, K., Yeseswini, K., Reddy, C. B., et al. (2025) · doi:10.1109/csitss67709.2025.11295680 · Cited by 2
“The text-as-data analytical approaches in political science receive complete assessment from Grimmer and Stewart [4] through their research study.”
Shankar, R., Devi, F., & Qian, X. (2025) · Biology Methods and Protocols · doi:10.1093/biomethods/bpaf088 · Cited by 3
Global China, Cooperative Migration Diplomacy and the Future of Transnational Labour
Kaneti, M., Hou, Y., & Israruddin (2026) · Population Space and Place · doi:10.1002/psp.70200 · Cited by 2
“This approach is in line with recent scholarship that treats computational methods as aids to interpretive social science rather than substitutes for it (Grimmer and Stewart 2013; Nelson 2020).”
Issue Attention in Public Opinion Polls: Pollsters as Agenda Responders and Agenda Setters
Zhang, Q., Witko, C., & Enns, P. K. (2026) · Policy Studies Journal · doi:10.1111/psj.70098 · Cited by 2
“There are many different approaches to automated content coding generally (Grimmer and Stewart 2013) and to policy content, specifically (Dun et al. 2021).”
Yin, L., Silverman, R. M., Wang, S., et al. (2026) · Transforming Government People Process and Policy · doi:10.1108/tg-09-2025-0272 · Cited by 2
Zhou, Q., Xu, K., & Acur, N. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103257 · Cited by 3
Equity How and Equity for Whom? Incorporating Equity Into Local Government Budgeting Processes
Kennedy, A. R., Kim, Y., & Moran, T. (2026) · Public Administration · doi:10.1111/padm.70049 · Cited by 4
Fighting for the Future: From Societalisation to Politicisation of Chile’s Dominga Mining Project
Undurraga, T., Fergnani, M., & Güell, P. (2026) · Cultural Sociology · doi:10.1177/17499755251409268 · Cited by 2
Sakamoto, T., Matsuoka, T., & Ito, H. (2026) · Journal of Peace Research · doi:10.1093/jopres/xjag018 · Cited by 3
Code and Creed: The Construction of AI‐Islamic Discourse in Singapore's Media Landscape
Shaker, R. A. (2026) · Journal for the Scientific Study of Religion · doi:10.1111/jssr.70056 · Cited by 2
Measuring issue salience for political parties using LLMs
Benoit, K. R., & Laver, M. (2026) · West European Politics · doi:10.1080/01402382.2026.2637134 · Cited by 2
Exploring structural features for position analysis in political discussions
Zirn, C., Schäfer, M., Strube, M., et al. (2014) · MADOC (University of Mannheim) · Cited by 2
Metadata
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
