Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Open accessAbstract
We provide a guide to automated content analysis methods for political texts, stressing that these methods reduce the costs of analyzing large corpora but require problem-specific validation.
AI snapshot
Written from the paper’s full text.
TL;DR
This article is a guide to automated content analysis for political texts, arguing that such methods can make large-scale text analysis feasible but are imperfect models of language that require problem-specific validation. It surveys classification, clustering/topic modeling, and scaling approaches, illustrating both successes and failures across political corpora.
Key findings
- Automated methods can reduce the cost of analyzing large text collections, but the authors stress that no single method is globally best and that all quantitative language models are wrong in some way.
- In a supervised classification example using 7,920 Russian public statements from 1998–2008, with 300 hand-coded documents and a Random Forest model under ten-fold cross-validation, overall accuracy was 0.65; for the “restrained” category, precision was 0.65 and recall was 0.75.
- In an unsupervised topic-model example using over 64,000 Senate press releases from 2005–2007, the expressed agenda model produced 44 topics; its credit-claiming measure correlated at 0.96 with a supervised ReadMe estimate.
- In scaling, wordfish performed well on German party platforms but failed to separate Democratic and Republican senators in Senate press releases; instead it recovered a nonideological position-taking versus credit-claiming spectrum, correlating at 0.86 with the expressed agenda measure.
Methods
- Design: a methodological review and practical guide organized around four principles: all quantitative language models are wrong but some are useful; methods augment rather than replace humans; there is no globally best method; and validation is essential.
- Data/sample: illustrative applications use Russian elite statements, over 64,000 Senate press releases from 2005–2007, German party platforms, and references to other political text corpora.
- Main analysis: the authors describe preprocessing text into document-term matrices, dictionary and supervised classification methods, fully automated and computer-assisted clustering, topic models, and supervised/unsupervised scaling methods, with validation strategies such as cross-validation, semantic and predictive validity checks, and convergent validity comparisons.
Limitations
- The paper states that automated methods are incorrect models of language, that performance cannot be guaranteed on new data, and that validation is therefore necessary; it also notes that dictionaries are often applied outside their original domain and are rarely validated.
- The authors state that unsupervised methods and scaling models require extensive substance-based validation, and that scaling methods rely on a strong ideological-dominance assumption that can fail in nonideological speech.
- Evident from the text: the survey is not exhaustive, excludes natural language processing and event-parsing methods, and the space given to methods is not proportional to their use among political scientists.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
12 of 89 cited sources checked · 12 supported · 0 flagged
- Premisecitation
Almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded.
“In fact, almost all classification methods implicitly assume that the training set is a random sample from the population of documents to be coded (Hand 2006).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Hand 2006 source
- Premisecitation
Providing guidance on when specific measures of similarity, objective functions, or optimization algorithms are likely to lead to useful or insightful partitions of data without specific knowledge of the data set may be impossible.
“Indeed, it appears that providing this guidance without specific knowledge of the data set may be impossible (Grimmer and King 2011).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Grimmer and King 2011 source
- Premisecitation
Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.
“Political scientists have extended topic models so that the parameters correspond to politically relevant quantities of interest.”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Quinn et al. 2010 source
- cites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
The dynamic multitopic model (Quinn et al. 2010) models daily attention to topics in Senate floor speeches.
“The first topic model in political science is the dynamic multitopic model (Quinn et al. 2010), used to model the daily attention to topics in Senate floor speeches”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Quinn et al. 2010 source
- Premisecitation
Grimmer and King (2011) show how to embed partitions into a two-dimensional space such that two clusterings are close if they organize documents in similar ways.
“Then, Grimmer and King (2011) show how to embed the partitions into a two-dimensional space such that two clusterings are close in the space if they organize the documents in similar ways.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Grimmer and King 2011 source
- Premisecitation
Unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.
“As Quinn et al. (2010) observe, unsupervised methods shift the user burden from determining categories before analysis to validating model output afterward.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Quinn et al. 2010 source
- Findingcitation
Consistently across applications, scholars have shown that a simple representation of text is sufficient to infer substantively interesting properties of texts.
“consistently across applications, scholars have shown that this simple representation of text is sufficient to infer substantively interesting properties of texts (Hopkins and King 2010).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingnone given
Sentences where word order fundamentally changes the nature of the sentence are empirically rare.
“empirically these sentences are rare.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Findingown data
The overall accuracy of the Random Forest classifier in the Russian military discourse example is 0.65.
“Accuracy ¼ No: Doc: On Diagonal Total No: Doc: ¼ 0:65:”
Quote in the paperNot a substantial claim - Findingown data
The recall for the restrained category is 0.75.
“For the restrained category, the recall is 0.75.”
Quote in the paperNot a substantial claim - Findingown data
The precision for the restrained foreign policy category is 0.65.
“For restrained foreign policy positions, the precision is 0.65.”
Quote in the paperNot a substantial claim - Findingown data
Both plots in Fig. 4 show that external events predict spikes in attention to press releases.
“Both plots in Fig. 4 show that external events predict spikes in attention to press releases.”
Quote in the paperSubstantial claim - Findingown data
When major figures die, such as Pope John Paul II, more honorary press releases are issued.
“Likewise, when major figures die, such as Pope John Paul II, more honorary press releases are issued.”
Quote in the paperSubstantial claim - Findingown data
When major events happen surrounding the war, such as the release of the Iraq study group report, more press releases are issued about Iraq.
“When major events happen surrounding the war-such as the release of the Iraq study group report-more press releases are issued about Iraq.”
Quote in the paperSubstantial claim - Findingown data
Figure 3 shows that the expressed agenda model applied to the press releases produces a higher quality clustering, more semantically valid, than any of the comparison methods on the same press releases.
“Figure 3 shows that the expressed agenda model applied to the press releases produces a higher quality clustering-more semantically valid-than any of the comparison methods on the same press releases.”
Quote in the paperSubstantial claim - Findingown data
Wordfish has identified an interesting, though nonideological, spectrum in the Senate press releases.
“Clear evidence that wordfish has identified this interesting-though nonideological-spectrum in the Senate press releases.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
- Findingown data
The ideological dominance assumption appears not to hold in Senate press releases, where senators regularly engage in nonideological credit claiming.
“we show below that the ideological dominance assumption appears to not hold in Senate press releases, where senators regularly engage in nonideological credit claiming.”
Quote in the paperSubstantial claim - Findingown data
For almost every topic committee pair in each year, leaders of Senate committees allocate substantially more attention to issues under their jurisdiction than other senators.
“For almost every topic committee pair in each year, leaders of Senate committees allocate substantially more attention to issues under their jurisdiction than other senators.”
Quote in the paperSubstantial claim - Findingown data
Figure 6 shows a strong correlation between the supervised and unsupervised methods.
“Figure 6 shows the strong correlation between the supervised and unsupervised methods.”
Quote in the paperNot a substantial claim - Findingown data
The gray points group around the 45 line, demonstrating that the unsupervised method is essentially equivalent to the supervised method.
“Figure 6 shows clearly that the gray points group around the 45 line, demonstrating that the unsupervised method is essentially equivalent to the supervised method.”
Quote in the paperSubstantial claim - Findingreasoning
When political actors are engaging in heavily ideological speech-as in German party platforms-unsupervised methods appear to retrieve reliable position estimates.
“When political actors are engaging in heavily ideological speech-as in German party platforms-unsupervised methods appear to retrieve reliable position estimates.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Actors' ideological leanings determine what is discussed in texts.
- Findingown data
The relationship between the Grimmer (2012) scaling and the wordfish scaling of Senate press releases is extremely strong, correlating at 0.86.
“The relationship between the two measures is extremely strong-correlating at 0.86.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
- Findingown data
This is true across all senators, resulting in a correlation of 0.96.
“This is true across all senators, resulting in a correlation of 0.96.”
Quote in the paperSubstantial claim - Findingcitation
The authors replicate an example from Slapin and Proksch (2008) showing that the ideological dominance model works well with German political platforms.
“we replicate an example from Slapin and Proksch (2008) that shows that the model works well with German political platforms.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Slapin and Proksch 2008 source
- Findingown data
Figure 5 shows that committee leaders allocate more attention to issues under their committee's jurisdiction than the average senator.
“Figure 5 shows that committee leaders allocate more attention to issues under their committee's jurisdiction than the average senator.”
Quote in the paperSubstantial claim - Findingreasoning
The wordfish algorithm applied to Senate press-release data fails to separate Democrat and Republican senators.
“The model clearly fails to separate Democrat and Republican senators-a necessity for any valid scaling in the now polarized Senate.”
Quote in the paperSubstantial claim- Rests on an assumption: The Senate is now polarized.
- Findingown data
When political actors can avoid ideological speech-as in Senate press releases-scaling methods retrieve some other, nonideological scaling.
“But when political actors can avoid ideological speech-as in Senate press releases-scaling methods retrieve some other, nonideological scaling.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Actors' ideological leanings determine what is discussed in texts.
- Findingown data
The unsupervised estimate of credit-claiming behavior is essentially equivalent to the estimate from ReadMe.
“The unsupervised estimate of credit-claiming behavior is essentially equivalent to the estimate from ReadMe.”
Quote in the paperSubstantial claim - Findingown data
When the ideological dominance assumption fails to fit the data, wordfish fails to retrieve underlying policy dimensions.
“But when ideological dominance assumption fails to fit the data, wordfish fails to retrieve underlying policy dimensions.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Actors' ideological leanings determine what is discussed in texts.
- Premisereasoning
The primary problem with using texts to make inferences about politics is volume: there are too many political texts.
“The primary problem is volume: there are simply too many political texts.”
Quote in the paperSubstantial claim - Interpretationreasoning
Analyzing massive collections of text has been essentially impossible for all but the most well-funded projects.
“analyzing massive collections of text has been essentially impossible for all but the most well-funded projects.”
Quote in the paperSubstantial claim - Interpretationreasoning
The performance of any one automated content method on a new data set cannot be guaranteed.
“the performance of any one method on a new data set cannot be guaranteed”
Quote in the paperNot a substantial claim - Interpretationreasoning
Automated content methods can make possible the systematic analysis of large-scale text collections without massive funding support.
“automated content methods can make possible the previously impossible in political science: the systematic analysis of large-scale text collections without massive funding support.”
Quote in the paperSubstantial claim - Interpretationcitation
Discarding word order would mask information on land negotiations in treaties between Native Americans and the U.S. government.
“In this case, discarding word order would mask information on land negotiations.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Interpretationcitation
Alternative model evaluations that rely on model fit or predicting the content of new texts can select substantively inferior models.
“Alternative model evaluations that rely on model fit or predicting the content of new texts can select substantively inferior models (see Chang et al. 2009).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Chang et al. 2009
- Interpretationreasoning
Human-based methods for making these inferences are both time and resource intensive.
“Human-based methods for making these inferences are both time and resource intensive.”
Quote in the paperSubstantial claim - Interpretationreasoning
Establishing one method to always use for one task is almost guaranteed to be impossible.
“establishing one method to always use for one task is almost guaranteed to be impossible.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Discarding text not related to the primary quantity of interest can improve the performance of automated clustering methods.
“discarding text not related to the primary quantity of interest can actually improve the performance of automated clustering methods.”
Quote in the paperNot a substantial claim - Interpretationnone given
Validation for unsupervised methods is less direct than for supervised methods.
“Validation for unsupervised methods is less direct.”
Quote in the paperSubstantial claim - Interpretationreasoning
For shorter texts, accompanying information or an extremely large volume of texts is often necessary for classification or scaling methods to perform reliably.
“For shorter texts, accompanying information (or an extremely large volume of texts) is often necessary for classification or scaling methods to perform reliably.”
Quote in the paperSubstantial claim - Interpretationreasoning
Each research question implies different models, or families of models, and different methods of validation.
“Each of the research questions imply different models, or families of models, to be used for analysis, and different methods of validation.”
Quote in the paperSubstantial claim - Interpretationreasoning
Both word scores and word fish rely heavily on the assumption that ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Ideology dominates the language used in the text.
- Interpretationreasoning
Including more realistic features into quantitative models does not necessarily translate into an improved method.
“Including more realistic features into quantitative models does not necessarily translate into an improved method”
Quote in the paperSubstantial claim - Interpretationreasoning
The goal of building text models is different than model building to make causal inferences.
“The goal of building text models is therefore different than model building to make causal inferences-the primary goal of most political science models.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The same model will perform well on some data sets but poorly when applied to other data.
“the same model will perform well on some data sets, but will perform poorly when applied to other data.”
Quote in the paperSubstantial claim - Interpretationreasoning
Lengthier texts, such as newspapers or party platforms, are often much easier to analyze than shorter statements, such as open-ended survey responses.
“This makes lengthier texts-newspapers or party platforms-often much easier to analyze than shorter statements, such as open-ended survey responses.”
Quote in the paperNot a substantial claim - Interpretationreasoning
When the assumption that ideology dominates language fits, the models can perform well.
“When this assumption fits, when political actors are expressing their policy positions in texts, the models can perform well.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Ideology dominates the language used in the text.
- Interpretationreasoning
Models that are less sophisticated in the use of language may provide more useful analysis of texts.
“models that are less sophisticated in the use of language may provide more useful analysis of texts.”
Quote in the paperNot a substantial claim - Interpretationreasoning
A simple list of words (unigrams) is often sufficient to convey the general meaning of a text.
“A simple list of words, which we call unigrams, is often sufficient to convey the general meaning of a text.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
- Interpretationreasoning
Automated methods can mitigate the cost of assigning documents to categories by limiting the amount of classification humans perform.
“Automated methods can mitigate the cost of assigning documents to categories, by limiting the amount of classification humans perform.”
Quote in the paperSubstantial claim - Interpretationreasoning
All methods necessarily fail to provide an accurate account of the data-generating process used to produce texts.
“The complexity of language implies that all methods necessarily fail to provide an accurate account of the data-generating process used to produce texts.”
Quote in the paperNot a substantial claim - Interpretationnone given
When applied to any one problem, the output of automated text models may be misleading or simply wrong.
“When applied to any one problem, however, the output of the models may be misleading or simply wrong.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Different data sets and different research questions often lead to different quantities of interest.
“Different data sets and different research questions often lead to different quantities of interest.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Dictionaries are relatively easy and cheap to apply across a variety of problems.
“The general idea of dictionaries make them relatively easy and cheap to apply across a variety of problems”
Quote in the paperNot a substantial claim0 of 3 sources checked - Interpretationreasoning
The same advice does not hold for quantitative text analysis methods.
“But the same advice does not hold for quantitative text analysis methods.”
Quote in the paperNot a substantial claim - Interpretationreasoning
When the assumption that ideology dominates language is violated, the model will place actors in a different and non-ideological space.
“When this assumption is violated, the model will place actors in a different and non-ideological space.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Ideology dominates the language used in the text.
- Interpretationnone given
The non-ideological space produced when the assumption is violated can be useful.
“We show that this space can be useful”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Ideology dominates the language used in the text.
- Interpretationreasoning
Automated content analysis methods use insightful but wrong models of political text to help researchers make inferences from their data.
“Automated content analysis methods use insightful, but wrong, models of political text to help researchers make inferences from their data.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Reducing the assumptions used may not imply more productive analyses.
“reducing the assumptions used may not imply more productive analyses.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Mosteller and Wallace's analysis of the Federalist Papers relied on information that is customarily discarded.
“Thus, their entire analysis relied on information that we customarily discard.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Mosteller and Wallace (1963)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
The ReadMe algorithm is not an option for all users, particularly those who have few documents or need to stratify results by an external covariate.
“But the ReadMe algorithm is not an option for all users, particularly those who have few documents or need to stratify the results by some external covariate.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Supervised learning is necessarily domain specific and therefore avoids the problems of applying dictionaries outside their intended area of use.
“First, it is necessarily domain specific and therefore avoids the problems of applying dictionaries outside of their intended area of use.”
Quote in the paperSubstantial claim - Interpretationreasoning
Applying dictionaries outside the domain for which they were developed can lead to inefficiencies in real data problems.
“This can lead to inefficiencies in applying dictionaries to real data problems-particularly if dictionaries are applied outside of the domain for which they were originally developed.”
Quote in the paperSubstantial claim - Interpretationcitation
ReadMe can provide reliable estimates of proportions across many domains and applications.
“The result is that ReadMe can provide reliable estimates of proportions across many domains and applications.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Many words that have a negative connotation in other contexts may have a positive connotation in corporate earning reports.
“They point out that many words that have a negative connotation in other contexts, like tax, cost, crude (oil), or cancer, may have a positive connotation in earning reports.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Loughran and McDonald (2011) source
- Interpretationcitation
When dictionaries are created in one substantive area and then applied to another, serious errors can occur.
“But when dictionaries are created in one substantive area and then applied to another, serious errors can occur.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Loughran and McDonald (2011) source
- Rests on an assumption: For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
- Interpretationreasoning
Supervised methods will never contribute a new coding scheme.
“But supervised methods will never contribute a new coding scheme.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The difficulty of deriving categories beforehand is due in part to the massive number of potential organizations of even a small number of texts.
“This difficulty is due in part to the massive number of potential organizations of even a small number of texts.”
Quote in the paperSubstantial claim - Premisecitation
Cross-validation can be used to replicate the ideal validation procedure.
“But cross-validation can be used to replicate this ideal procedure (Efron and Gong 1983;Hastie, Tibshirani, and Friedman 2001).”
Quote in the paperNot a substantial claim0 of 2 sources checked - Interpretationreasoning
Most analyses based on dictionaries are built on shaky foundations.
“The consequence of domain specificity and lack of validation is that most analyses based on dictionaries are built on shaky foundations.”
Quote in the paperSubstantial claim- Rests on an assumption: Measures from dictionaries are rarely validated.
- Rests on an assumption: Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
- Interpretationreasoning
Training sets need enough documents to apply supervised methods accurately.
“Training sets also need enough documents to apply supervised methods accurately.”
Quote in the paperNot a substantial claim - Premisecitation
As long as classifiers are accurate and diverse, combining them will improve accuracy.
“Heuristically, as long as the classifiers are accurate and diverse, combining the classifiers will improve accuracy (Jurafsky and Martin 2009).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Humans are unable to produce the same granular measures reliably.
“Although this is useful for applications, humans are unable to produce the same granular measures reliably (Krosnick 1999).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Krosnick 1999 source
- Interpretationreasoning
Best performance comes from random sampling to obtain a representative sample of documents.
“Given this assumption, it is not surprising that best performance comes from random sampling to obtain a representative sample of documents-either through simple random sampling or from a more complicated stratified design.”
Quote in the paperSubstantial claim - Premisecitation
Words not identified as negative in off-the-shelf dictionaries may have quite negative connotation in earning reports.
“And words that are not identified as negative in off-the-shelf dictionaries may have quite negative connotation in earning reports (e.g., unanticipated).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Loughran and McDonald (2011) source
- Premisecitation
Shifting focus to estimating proportions can lead to substantial improvements in accuracy even if documents are not randomly sampled from the corpus.
“Shifting focus to estimating proportions, PðCÞ, can lead to substantial improvements in accuracy-even if the documents are not randomly sampled from the corpus (Hopkins and King 2010).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hopkins and King 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationcitation
Combining classification models can produce a classifier superior to any of the individual classifiers.
“But it is also straightforward to combine classification models to produce a classifier which is superior to any of the individual classifiers.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Supervised learning methods provide a useful alternative method for assigning documents to predetermined categories.
“Supervised learning methods provide a useful alternative method for assigning documents to predetermined categories.”
Quote in the paperSubstantial claim - Interpretationreasoning
The set of categories may be difficult to derive beforehand in some instances.
“In other instances, however, the set of categories may be difficult to derive beforehand.”
Quote in the paperNot a substantial claim - Interpretationreasoning
As the number of categories in a coding scheme increases, the number of documents needed in the training set also increases.
“For example, as the number of categories in a coding scheme increases, the number of documents needed in the training set also increases.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Enumerating all possible partitions of a collection of texts is impossible.
“But enumerating these partitions is impossible.”
Quote in the paperSubstantial claim - Interpretationnone given
For even moderately sized data sets, the number of potential partitions is much larger than the estimated number of atoms in the universe.
“For even moderately sized data sets, say one hundred documents, the number of potential partitions is much larger than the estimated number of atoms in the universe.”
Quote in the paperNot a substantial claim - Interpretationnone given
If scholars use dictionaries to code documents into binary categories, then validation based on human gold standards is straightforward.
“If scholars use dictionaries to code documents into binary categories (e.g., positive or negative tone), then validation based on human gold standards and the methods we describe in Section 5.2.3 is straightforward.”
Quote in the paperNot a substantial claim - Interpretationreasoning
It is essentially impossible to derive gold-standard evaluations of dictionaries based on human coding of documents.
“The result is that it is essentially impossible to derive gold-standard evaluations of dictionaries based on human coding of documents.”
Quote in the paperSubstantial claim - Interpretationcitation
Naive Bayes can capture information in texts useful for classification even though its conditional independence assumption is wrong.
“But even though the assumption is wrong, the model is still able to capture information in texts useful for classification.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hastie, Tibshirani, and Friedman 2001 source
- Premisecitation
There is often a negative relationship between the best-fitting model and the substantive information provided.
“We also cannot turn to fit statistics, as Chang et al. (2009) show that there is often a negative relationship between the best-fitting model and the substantive information provided.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Chang et al. 2009
- Premisenone given
Unsupervised methods for content analysis reduce the information in large text collections substantially.
“Unsupervised methods for content analysis reduce the information in large text collections substantially.”
Quote in the paperNot a substantial claim - Interpretationreasoning
CAC provides a way to identify new or understudied concepts even in already extensively analyzed texts.
“First, the method provides a way to identify new or understudied concepts-even in already extensively analyzed texts.”
Quote in the paperSubstantial claim - Premisecitation
Models that estimate the number of clusters are heavily model dependent.
“Models that estimate the number of clusters are heavily model dependent (Wallach et al. 2010).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wallach et al. 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Nonparametric models, such as the Dirichlet process prior, still make model-based decisions on the number of clusters or topics to include.
“Nonparametric models, such as the Dirichlet process prior, still make model-based decisions on the number of clusters or topics to include.”
Quote in the paperSubstantial claim - Premisecitation
Grimmer (2012) shows that where senators fall on the spectrum of expressed priorities has real consequences for representation.
“And Grimmer (2012) shows that where senators fall on this spectrum has real consequences for representation.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
CAC provides a way to evaluate the originality of new clustering methods.
“Third, CAC provides a way to evaluate the originality of new clustering methods.”
Quote in the paperNot a substantial claim - Premisenone given
Each clustering method relies on a set of assumptions that result in varying performance in different contexts.
“While each of the clustering methods are well specified and based on careful derivation, each method relies on a set of assumptions that result in varying performance in different contexts.”
Quote in the paperNot a substantial claim - Premisecitation
Applying the unsupervised model leads to an understudied organization of senators, based on how they present their work to constituents.
“Applying the unsupervised model leads to an understudied organization of senators, based on how they present their work to constituents.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Recent studies show that the estimated number of clusters is strongly model dependent.
“but recent studies show that the estimated number of clusters is strongly model dependent (Wallach et al. 2010).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Wallach et al. 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisenone given
Both CAC and FAC methods require setting the number of clusters in a model.
“Both CAC and FAC methods require setting the number of clusters in a model.”
Quote in the paperNot a substantial claim - Interpretationcitation
When the ideological dominance assumption fits the data, the model can reliably retrieve valid ideological dimensions from political texts.
“When the ideological dominance assumption fits the data the model can reliably retrieve valid ideological dimensions from political texts.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Actors' ideological leanings determine what is discussed in texts.
- Interpretationown data
It provides one direct test to ensure that the output from an unsupervised method is just as valid, reliable, and useful as the categorization schemes from supervised methods.
“It does, however, provide one direct test to ensure that the output from an unsupervised method is just as valid, reliable, and useful as the categorization schemes from supervised methods.”
Quote in the paperSubstantial claim - Interpretationreasoning
Both the strength and limitations of IRT methods for scaling are the lack of supervision.
“Both the strength and limitations of IRT methods for scaling are the lack of supervision.”
Quote in the paperSubstantial claim - Premisereasoning
Unsupervised scaling methods discover words that distinguish locations on a political spectrum rather than rely on reference texts.
“Rather than rely on reference texts for scaling documents, unsupervised scaling methods discover words that distinguish locations on a political spectrum.”
Quote in the paperSubstantial claim - Interpretationreasoning
This implies that all the confidence we would attach to using the estimates from the supervised method can also be attached to the unsupervised method.
“This implies that all the confidence we would attach to using the estimates from the supervised method can also be attached to the unsupervised method.”
Quote in the paperSubstantial claim - Premisenone given
Other political actors, presidents, bureaucrats, and political candidates, do not cast votes.
“And other political actors-presidents, bureaucrats, and political candidates-do not cast votes.”
Quote in the paperSubstantial claim - Premisecitation
Outside the U.S. Congress, roll call votes are less reliable.
“but outside the U.S. Congress roll call votes are less reliable (Spirling and McLean 2007).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Spirling and McLean 2007 source
- Premisecitation
One method for scaling political actors using texts, based on Laver, Benoit, and Garry (2003), is a supervised method, analogous to dictionary methods, to situate actors in space based on their words.
“One method, based on Laver, Benoit, and Garry (2003), is a supervised method-analogous to dictionary methods-to situate actors in space based on their words.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Laver, Benoit, and Garry 2003 source
- Premisecitation
Monroe and Maeda (2004) and later Slapin and Proksch (2008) introduced statistical models based on item response theory (IRT) to automatically estimate the spatial location of parties.
“First Monroe and Maeda (2004), then later Slapin and Proksch (2008), introduce statistical models based on item response theory (IRT) to automatically estimate the spatial location of the parties.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Monroe and Maeda 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Slapin and Proksch 2008 source
- Premisenone given
Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
Quote in the paperSubstantial claim - Interpretationreasoning
Scaling methods will have more even performance across texts if they are accompanied with methods that separate ideological and non-ideological statements.
“Scaling methods will have more even performance across texts if they are accompanied with methods that separate ideological and non-ideological statements.”
Quote in the paperSubstantial claim - Premisecitation
Beauchamp (2011) shows that output from text scaling methods can be used to predict votes in Congress.
“Beauchamp (2011), for example, shows that the output from text scaling methods can be used to predict votes in Congress.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
Careful preprocessing of texts to remove words that are likely only stylistic can mitigate part of the dependence of wordscores on reference texts.
“Careful preprocessing of texts, to remove words that are likely only stylistic, can mitigate part of this problem.”
Quote in the paperSubstantial claim - Interpretationreasoning
No amount of preprocessing can completely eliminate the dependence of wordscores on reference texts.
“But no amount of preprocessing can completely eliminate it.”
Quote in the paperSubstantial claim - Interpretationown data
This suggests that the measures are, at least by this test, predictively valid.
“This suggests that the measures are, at least by this test, predictively valid.”
Quote in the paperSubstantial claim - Interpretationreasoning
If the goal is to replicate expert opinion or already existent scalings, then text methods are unnecessary.
“Certainly plausibility of measures is important, but if the goal is to replicate expert opinion, or already existent scalings, then text methods are unnecessary.”
Quote in the paperNot a substantial claim - Premisecitation
Recent papers have implicitly equated the goal of scaling methods as replicating expert opinion or well-validated scalings made using nontext data.
“Recent papers have implicitly equated the goal of scaling methods as replicating expert opinion (Benoit, Laver, and Mikhaylov 2009;Mikhaylov, Laver, and Benoit 2010) or well-validated scalings made using nontext data (Beauchamp 2011).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Benoit, Laver, and Mikhaylov 2009 source
- cites Mikhaylov, Laver, and Benoit 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Grimmer (2012) shows that several of the topics identify press releases that are claiming credit for money directed to the district.
“Grimmer (2012) shows that several of the topics identify press releases that are claiming credit for money directed to the district (Mayhew 1974) (the differences across topics capture differences in the type of money claimed).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Mayhew 1974
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
The lack of supervision and the use of an IRT model imply that the model will seize upon the primary variation in language across actors.
“But the lack of supervision and the use of an IRT model implies that the model will seize upon the primary variation in language across actors.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- Premisecitation
Other methods for scaling political actors rely on particular disclosure institutions that are often absent in other democracies.
“Other methods for scaling political actors have been developed (e.g., Gerber and Lewis 2004;Bonica 2011), but they rely on particular disclosure institutions that are often absent in other democracies.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Gerber and Lewis 2004 source
- cites Bonica 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Beauchamp (2011) shows that the ideological dominance assumption works well in Senate floor speeches.
“For example, Beauchamp (2011) shows that this works well in Senate floor speeches”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
Because the model does not include supervision explicitly, it is difficult to guarantee that the output of the model will reliably identify the revealed ideological locations of political actors.
“Because the model does not include supervision explicitly, it is difficult to guarantee that the output of the model will reliably identify the revealed ideological locations of political actors.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- Premisecitation
A second method for scaling political actors using texts is an unsupervised method for locating actors in space.
“A second method is an unsupervised method for locating actors in space (Monroe and Maeda 2004;Slapin and Proksch 2008).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Monroe and Maeda 2004
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Slapin and Proksch 2008 source
- Interpretationnone given
When the IRT model works well it provides reliable estimates of political actors' spatial locations with little resource investment.
“When the model works well it provides reliable estimates of political actors' spatial locations with little resource investment.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- Interpretationreasoning
Simple extrapolation from experts or existing scaling would suffice if the goal is to replicate expert opinion or existing scalings.
“Simple extrapolation from the experts or existing scaling would suffice.”
Quote in the paperNot a substantial claim
- Classified as probably not a substantial claim. 158
- May combine two claims. 34
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationOnline platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient.
“online platforms that distribute tasks across workers, such as Amazon's Mechanical Turk, can make acquiring data efficient (Berinsky, Huber, and Lenz 2012).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Berinsky, Huber, and Lenz 2012 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMonroe, Colaresi, and Quinn (2008) describe methods that measure how well words separate already identified categories of interest.
“Monroe, Colaresi, and Quinn (2008) describe a variety of methods that measure how well words separate already identified categories of interest”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Monroe, Colaresi, and Quinn (2008) source
May combine two claims.
PremisecitationUsing experimentally elicited human input allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.
“Using this method allows direct comparison of the quality of clusterings, but it is unable to provide an absolute measure of semantic validity.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Grimmer and King 2011 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGrimmer and King (2011) introduce a method for exploring the space, easily searching over included methods and millions of other clusterings.
“Using this space, Grimmer and King (2011) introduce a method for exploring it, easily searching over the included methods and millions of other clusterings that are the result of combinations of similar organizations of the data.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Grimmer and King 2011 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCAC is a method for efficiently searching over a large space of clusterings.
“With this idea in mind, CAC is a method for efficiently searching over a large space of clusterings (Grimmer and King 2011).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Grimmer and King 2011 source
Classified as probably not a substantial claim.
PremisecitationIf topics are valid, then external events should explain sudden increases in attention to a topic.
“Quinn et al. (2010) argue that if topics are valid, then external events should explain sudden increases in attention to a topic”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Quinn et al. 2010 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenFor a moderate volume of documents without a particularly specialized vocabulary, the document-term matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes.
“For a moderate volume of documents without a particularly specialized vocabulary, this matrix will have between three thousand and five thousand features or terms and will contain mostly zeroes (a condition we call sparsity).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
FindingreasoningThe topics in Table 4 exemplify the utility of unsupervised learning.
“The topics in Table 4 exemplify the utility of unsupervised learning.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe model also identifies a topic of press releases that honor constituents, commemorating a national holiday or a tribute to a deceased constituent.
“But the model also identifies a topic of press releases that honor constituents-commemorating a national holiday or a tribute to a deceased constituent.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe first row of Table 4 identifies a topic about the Iraq war.
“The first row of Table 4 identifies a topic about the Iraq war-one of the most salient debates during the time the press releases were issued.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).
“The final row of Table 4 is a topic about claiming credit for grants allocated to fire departments through the Assistance to Firefighter Grant Program (AFGP).”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Findingown dataThis includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.
“This includes clusterings from the nonparametric method and clusterings produced by two Senate press secretaries.”
Quote in the paperSubstantial claimMay combine two claims.
FindingreasoningThe wordfish scaling of Senate press releases is meaningful substantively but does not correspond to standard policy space.
“The wordfish scaling is meaningful substantively, but it does not correspond to standard policy space.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Findingnone givenThe non-ideological locations that wordfish identifies are quite useful.
“In fact, we will show below that non-ideological locations that wordfish identifies are quite useful.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingcitationThe wordfish estimates in the left-hand plot of Fig. 7 separate the German parties and replicate expert assessments of German party movement over time.
“As Slapin and Proksch (2008) show, the estimates in the left-hand plot separate the German parties and replicate expert assessments of German party movement over time.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Slapin and Proksch 2008 source
- The authors state this as an assumption: Politicians reside in a low-dimensional political space.
- The authors state this as an assumption: A politician's or party's position in the political space affects the rate at which words are used in texts.
- The authors state this as an assumption: Each word j from individual i is drawn from a Poisson distribution with rate ij.
- The authors state this as an assumption: The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).
Premisenone given§ 3 Acquiring TextAutomated methods require texts stored in a plain text format.
“Automated methods require texts stored in a plain text format-UTF-8 for Latin characters, for example.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Premisenone given§ 1 IntroductionLanguage is the medium for politics and political conflict.
“Language is the medium for politics and political conflict.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenPolitics and political conflict often occur in the written and spoken word.
“Politics and political conflict often occur in the written and spoken word.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningValidation is essential when applying automated content methods.
“validation is essential when applying automated content methods.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenAutomated text analysis substantially reduces the costs of analyzing large collections of text.
“it substantially reduces the costs of analyzing large collections of text.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenApplying dictionaries outside the domain for which they were developed can lead to serious errors.
“applying dictionaries outside the domain for which they were developed can lead to serious errors.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningTo understand what politics is about we need to know what political actors are saying and writing.
“to understand what politics is about we need to know what political actors are saying and writing.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningValidation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.
“validation of a classifier's accuracy is essential to establish the reliability of supervised learning methods.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenAutomated methods require extensive and problem-specific validation.
“require extensive and problem-specific validation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenAutomated methods are no substitute for careful thought and close reading.
“they are no substitute for careful thought and close reading”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThe performance of any one classifier can vary substantially across contexts.
“the performance of any one classifier can vary substantially across contexts”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenHiring coders to manually read all documents is still very expensive.
“And hiring coders to manually read all documents is still very expensive.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningSupervised methods improve upon dictionaries.
“One way to improve upon dictionaries are supervised methods”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
PremisereasoningRarely are scholars able to manually read all the texts in even moderately sized corpora.
“Rarely are scholars able to manually read all the texts in even moderately sized corpora.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenThe massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.
“the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
PremisereasoningAutomated content methods are incorrect models of language.
“automated content methods are incorrect models of language.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningThe complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.
“the complexity of language implies that automated content analysis methods will never replace careful and close reading of texts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenFinding the separating words for dictionaries is relatively easy.
“Finding the separating words is also relatively easy.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenMost methods begin with a series of preprocessing steps to reduce the diversity of language to a manageable set of features.
“most methods begin in the same way with a series of preprocessing steps to reduce the awe inspiring diversity of language to a manageable set of features.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningValidation is essential for unsupervised methods.
“validation is essential.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAutomated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.
“Automated text analysis methods can substantially reduce the costs and time of analyzing massive collections of political texts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
InterpretationreasoningWithout a broader set of tools available, it is difficult to know if the methods included in commercially available software are optimal for the particular problem at hand.
“without a broader set of tools available, it is difficult to know if the methods included in the commercially available software are optimal for the particular problem at hand.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenTone is just one type of analysis a dictionary method can perform.
“Tone is just one type of analysis a dictionary method can perform.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningSupervised learning methods replicate the familiar manual coding task, but with a machine.
“Supervised learning methods replicate the familiar manual coding task, but with a machine.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningIt is often difficult, and sometimes impossible, to validate the output of commercial tools.
“It is often difficult, and sometimes impossible, to validate the output.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenThe recipe described above is one way to represent text as data.
“The recipe described above is one way to represent text as data.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenEven after coding rules are developed and coders are trained, manual methods require that coders read each individual text.
“Even after coding rules are developed and coders are trained, manual methods require that coders read each individual text.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningOnce a problem is identified in commercial tools, it is exceedingly difficult to change how the analysis is conducted.
“once a problem is identified, it is exceedingly difficult to change how the analysis is conducted.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningA deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.
“a deep understanding of the texts is one of the key advantages of the social scientist in applying automated methods.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe alternative strategies for representing text are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams).
“Although these are fundamentally similar ways of describing the same basic feature set (the list of unordered unigrams)”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenIt is difficult to know a priori if any one FAC method will provide useful clusterings in a particular context.
“it is difficult to know a priori if any one method will provide useful clusterings in a particular context.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningComputers amplify human abilities rather than replace humans.
“Rather than replace humans, computers amplify human abilities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThere is substantial variation within families of models.
“There is also substantial variation within families of models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningValidations are necessary to establish the meaning of the non-ideological space.
“validations are necessary to establish its meaning.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Ideology dominates the language used in the text.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Premisenone givenThe data generation process for any text is a mystery even to linguists.
“The data generation process for any text (including this one) is a mystery-even to linguists.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningThe majority of applied research uses stemming.
“the majority of applied research uses stemming.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationcitationDiscarding the order in which words occur in documents is the most consequential and shocking step.
“The most consequential, and shocking, step we will take is to discard the order in which words occur in documents (Jurafsky and Martin 2009).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Jurafsky and Martin 2009
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationcitationSometimes the problem calls for a completely different approach to representing text.
“sometimes the problem calls for a completely different approach.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenThe preprocessing steps are designed to retain information used by automated methods while discarding information likely to be unhelpful, ancillary, or too complex for a statistical model.
“Each of the steps are designed to retain information that will be used by the automated methods, while discarding information that will likely be unhelpful, ancillary, or too complex for use in a statistical model.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningThe stemmer is a much cruder algorithm than a lemmatizer but considerably faster.
“The stemmer is a much cruder algorithm, but considerably faster.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe performance of stemming does not seem to matter for most applications.
“The performance does not seem to matter for most applications”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationAutomated scraping methods make acquiring text data stored on Web sites easier.
“automated scraping methods make acquiring these data easier (Jackman 2006).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Jackman 2006
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
InterpretationreasoningCommercial tools for quantitative text analysis simply provide the researcher with output.
“These programs simply provide the researcher with output.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningApplying FAC algorithms to texts may have mixed results.
“Applying FAC algorithms to texts may have mixed results”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationScaling performed on the resulting matrix of distances is able to account for word order.
“Thus, scaling performed on the resulting matrix of distances is able to account for word order.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Spirling (2012) source
- The authors state this as an assumption: Documents are a bag of words, where word order does not inform analyses.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningMuch of the across-approach debate can be resolved by acknowledging that there are different research questions and designs that imply different types of models will be useful.
“much of the across-approach debate can be resolved simply by acknowledging that there are different research questions and designs that imply different types of models will be useful.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationUsing a high-quality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts.
“using a highquality scanner and Optical Character Recognition software, it is possible to convert archival materials into computer readable texts (see e.g., the data collection process in Eggers and Hainmueller 2009).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Eggers and Hainmueller 2009 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe recipe for representing text quantitatively is easy to apply with freely available software.
“The recipe is easy to apply, with several pieces of freely available software able to apply the sequence of steps we describe here.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAutomated methods rely on having a sufficient number of words to use reliably.
“Automated methods also rely on having a sufficient number of words to use reliably.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationThe Porter stemming algorithm is commonly employed because of its moderate approach to word simplification.
“the Porter stemming algorithm (Porter 1980) is commonly employed because of its moderate approach to word simplification.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Porter 1980 source
Classified as probably not a substantial claim. May combine two claims.
InterpretationreasoningResearchers still guide the process, make modeling decisions, and interpret the output of the models.
“researchers still guide the process, make modeling decisions, and interpret the output of the models.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationcitationDictionary methods are perhaps the most intuitive and easy to apply automated method.
“We begin with dictionary methods, perhaps the most intuitive and easy to apply automated method (Stone et al. 1966).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Stone et al. 1966
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThere is no guarantee that unsupervised methods will return classes that are theoretically interesting.
“there is no guarantee that unsupervised methods will return classes that are theoretically interesting”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningGuiding the process, making modeling decisions, and interpreting output require close reading of texts and thoughtful analysis by the researcher.
“All these require the close reading of texts and thoughtful analysis by the researcher.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAssigning texts to categories is the most common use of content analysis methods in political science.
“Assigning texts to categories is the most common use of content analysis methods in political science.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenThe goal of assigning texts to categories is to infer either the category of each document, the overall distribution of documents across categories, or both.
“In each instance, the goal is to infer either the category of each document, the overall distribution of documents across categories, or both.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenNot all of language's complexity is necessary to effectively analyze texts.
“not all of language's complexity is necessary to effectively analyze texts.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe usual advice for causal inference model building is that it is essential to include all the relevant features of the data-generating process.
“The usual advice for causal inference model building is that it is essential to include all the relevant features of the datagenerating process-either in covariates or in the model structure (e.g., Greene 2007).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Greene 2007
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningAutomated methods will not eliminate the need for careful thought by researchers nor remove the necessity of reading texts.
“These methods will not, however, eliminate the need for careful thought by researchers nor remove the necessity of reading texts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAutomated content methods tend to work better when the text is focused on the expression of one idea for classification or the expression of policy positions for scaling methods.
“These methods tend to work better when the text is focused-either on the expression of one idea for classification or the expression of policy positions for scaling methods.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenSupervised learning methods are much easier to validate than dictionary methods, with clear statistics that summarize model performance.
“Second, supervised learning methods are much easier to validate, with clear statistics that summarize model performance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningIf researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.
“If, however, researchers approach a problem without a predetermined categorization scheme, unsupervised methods can be useful.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningUsing an unsupervised learning method to place documents into predetermined categories is at best frustrating, particularly if the categories capture subtle differences in tone or sentiment.
“Using an unsupervised learning method to accomplish this task is at best frustrating-particularly if the predetermined categories are intended to capture subtle differences in tone or sentiment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningValidating a clustering is essential for demonstrating that the output of an unsupervised method is useful.
“Before using a clustering, validating the clustering is essential for demonstrating that the output of an unsupervised method is useful.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationCross-validation avoids overfitting by focusing on out-of-sample prediction and selects the best model from a set of candidate models.
“Cross-validation is extremely powerful-it avoids overfitting by focusing on out-of-sample prediction and selects the best model for the underlying data from a set of candidate models (this is known as the Oracle property) (van der Vaart, Dudoit, and van der Laan 2006).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites van der Vaart, Dudoit, and van der Laan 2006 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationK-Means is perhaps the most widely used FAC method.
“To build intuition, we introduce in detail the K-Means algorithm-perhaps the most widely used FAC method (MacQueen 1967).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites MacQueen 1967
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningSupervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.
“Far from competitors, supervised and unsupervised methods are most productively viewed as complementary methods, particularly for new projects or recently collected data sets.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataValidations of unsupervised methods are done conditional on the clustering produced; without first seeing the clustering, assessing validity is impossible.
“As we show in Section 6.4, validations are done conditional on the clustering produced: without first seeing the clustering, assessing validity is impossible.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationEnsembles are useful for increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers.
“But ensembles are also useful for other reasons, including: increased out-of-sample stability and the ability to capture complex functional forms with relatively simple classifiers (Dietterich 2000;Hillard, Purpura, and Wilkerson 2008).”
Quote in the paperNot a substantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationHopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.
“Hopkins and King (2010) offer five hundred as a rule of thumb with one hundred documents probably being enough.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationNaive Bayes has proven to be a useful classifier for a diverse set of tasks.
“The Naı¨ve Bayes's classifier conforms neatly to our first principle: although the model is clearly wrong-of course features are not conditionally independent-it has proven to be a useful classifier for a diverse set of tasks (Hastie, Tibshirani, and Friedman 2001).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hastie, Tibshirani, and Friedman 2001 source
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenDictionary methods require scholars to identify words that separate classes beforehand.
“Dictionary methods require scholars to identify words that separate classes beforehand.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationreasoningUnsupervised methods provide insights into classifications that would be difficult or impossible to obtain without guidance.
“In this case, unsupervised methods provide insights into classifications that would be difficult (or impossible) to obtain without guidance.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenAmbiguities in language, limited attention of coders, and nuanced concepts make reliable classification of documents difficult even for expert coders.
“Ambiguities in language, limited attention of coders, and nuanced concepts make the reliable classification of documents difficult-even for expert coders.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenEach FAC algorithm has three components: a definition of document similarity or distance, an objective function, and an optimization algorithm.
“The FAC literature is massive, but each algorithm has three components: (1) a definition of document similarity or distance; (2) an objective function that operationalizes an ideal clustering; and (3) an optimization algorithm.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenNo statistical model can repair a poorly constructed training set.
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenCompletely resolving the necessary model dependence in unsupervised models is impossible.
“Completely resolving the necessary model dependence in unsupervised models is impossible.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThe move to proportions in Hopkins and King (2010) weakens assumptions used in other methods and reduces potential bias systemic in other methods.
“The move to proportions in Hopkins and King (2010) pays substantial dividends for inference-weakening assumptions used in other methods and reducing the potential bias systemic in other methods.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hopkins and King (2010)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningAny one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.
“Any one of these methods could be used to produce dictionary-like scores of words, which could then be applied in other contexts to classify documents.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningThe number of documents necessary for a training set depends upon the specific application of interest.
“The number necessary will depend upon the specific application of interest.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationcitationIf scholars have documents already coded into categories, dictionaries can be produced using existing methods.
“if scholars have documents already coded into categories, dictionaries can be produced using existing methods.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe need to validate clusterings does not negate the value of unsupervised methods, nor does it make them a special case of supervised learning methods.
“The need to validate clusterings does not negate the value of unsupervised methods, nor does it lead to them becoming a special case of supervised learning methods (as suggested in Hillard, Purpura, and Wilkerson 2008).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Hillard, Purpura, and Wilkerson 2008 source
Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
Interpretationnone givenThe categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.
“The categories of interest in a new project or a new corpus are usually unclear or could benefit from extensive exploration of the data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationStewart and Zhukov (2009) show that Russian military elites are more activist in considering the use of force than their political counterparts, in contrast to conventional wisdom.
“Stewart and Zhukov (2009) use similar results to show that Russian military elites are actually more activist in considering the use of force than their political counterparts, in contrast to the conventional wisdom (Gelpi and Feaver 2002).”
Quote in the paperNot a substantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningThe actual properties of dictionary measures and how they relate to the concepts they attempt to measure are essentially a mystery.
“Yes, dictionaries are able to produce measures that are claimed to be about tone or emotion, but the actual properties of these measures-and how they relate to the concepts they are attempting to measure-are essentially a mystery.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone givenFAC methods are useful but necessarily model dependent.
“FAC methods are useful, but are necessarily model dependent.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenWell-coded documents can hide faults in simple models.
“The most important step in applying a supervised learning algorithm is constructing a reliable training set, because no statistical model can repair a poorly constructed training set, and wellcoded documents can hide faults in simple models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe debate casting unsupervised and supervised methods as competitor methods is misplaced; they are different models with different objectives.
“This debate is misplaced: supervised and unsupervised methods are different models with different objectives.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe recall rate is higher than precision because the Random Forest algorithm guesses too often that a document is restrained.
“The recall rate is higher than the precision here because the Random Forest algorithm guesses too often that a document is restrained.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe authors performed a ten-fold cross-validation using the training data to assess how well the Random Forest algorithm replicated human coders.
“To assess how well the Random Forest algorithm was able to replicate human coders, we performed a ten-fold cross-validation using the training data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningUnsupervised methods are valuable because they can identify organizations of text that are theoretically useful but perhaps understudied or previously unknown.
“Unsupervised methods are valuable because they can identify organizations of text that are theoretically useful, but perhaps understudied or previously unknown.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationFAC methods appear in an extensive literature across numerous fields, where they are used to estimate categories from data.
“This class of methods appears in an extensive literature across numerous fields, where they are used to estimate categories from data (for a review, see Manning, Raghavan, andSchu¨tze 2008 andJain, Murty, andFlynn 1999).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Manning, Raghavan, andSchu¨tze 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Jain, Murty, andFlynn 1999
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
InterpretationreasoningAll clustering methods are based on incorrect models of language, and a priori it is hard to know if any one method will provide substantively useful clusterings.
“All the clustering methods are based on incorrect models of language and a priori it is hard to know if any one method will provide substantively useful clusterings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe conditional independence assumption in Naive Bayes is wrong because word use is highly correlated in any data set.
“Of course, this assumption must be wrong: the use of words is highly correlated any data set.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: Given a document's category, the words are generated independently.
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenSupervised and dictionary methods assume a well-defined set of categories.
“Supervised and dictionary methods assume a well-defined set of categories.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningIf there are more than 100 documents for each value of a covariate, the ReadMe algorithm can be rerun on each strata; otherwise individual classification is necessary.
“If there are a large number of documents (in excess of 100 at least) for each value of the covariate, the algorithm can simply be rerun on each strata, but otherwise it is necessary to return to individual classification.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenDetermining the number of clusters is one of the most difficult questions in unsupervised learning.
“Determining the number of clusters is one of the most difficult questions in unsupervised learning.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisereasoningMeasures of statistical fit rely on the assumption that the goal is to model well the representation of texts after preprocessing.
“But this relies on the assumption that the goal is to model well the representation of texts after preprocessing.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe standard K-means optimization algorithm only guarantees convergence to a local minimum.
“This algorithm for optimization-closely related to the Expectation-Maximization algorithm-only guarantees convergence to a local minimum (Dempster, Laird, and Rubin 1977;Bishop 2006)”
Quote in the paperNot a substantial claim0 of 2 sources checkedMay combine two claims.
PremisecitationGrimmer (2012) shows that the spectrum of expressed priorities approximates spectra suggested in earlier qualitative work and predicted in formal theoretic models of Congress and Congressional elections.
“Grimmer (2012) shows that this spectrum is theoretically interesting: it approximates spectra suggested in earlier qualitative work (Fenno 1978;Yiannakis 1982), and predicted in formal theoretic models of Congress and Congressional elections (Weingast, Shepsle, and Johnsen 1981;Ashworth and Bueno de Mesquita 2006).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Fenno 1978
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Yiannakis 1982 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Premisenone givenThe K-means objective function is multimodal and non-continuous.
“the K-means' objective function is multimodal and non-continuous.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe expressed agenda model (Grimmer 2010) measures the attention senators allocate to press releases.
“Following Quinn et al. (2010) is the expressed agenda model (Grimmer 2010), which measures the attention senators allocate to press releases”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningIf a clustering method is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.
“If it is useful, then the distinct clusterings should lead to new or useful organizations of the documents that other methods miss.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationSome senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.
“Some senators allocate substantial attention to articulating positions, others allocate much more attention to credit claiming, and still others adopt a more evenly balanced presentational style.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationTopic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories.
“Topic models are a broad class of Bayesian generative models that encode problem-specific structure into an estimation of categories (see Blei 2012 for a review).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Blei 2012 source
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenIf a clustering method is truly new, then it should occupy a distinct part of the space, at least for some collections.
“If a clustering method is truly new, then it should occupy a distinct part of the space-at least for some collections.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningSeveral random restarts are often necessary to find the global optimum in K-means.
“and therefore several random restarts are often necessary to find the global optimum.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationGrimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.
“Grimmer (2012) shows that the primary variation underlying senators' expressed priorities is how senators balance position taking and credit claiming in their press releases.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone givenTopic models share two broad characteristics: a definition of a topic and a basic hierarchical structure.
“Topic models share two broad characteristics.The first is a definition of a topic.”
- Also stated in: § 6.1.2 Mixed membership models · plain
“Second, the models share a basic hierarchical structure.”
Quote in the paperNot a substantial claim- Also stated in: § 6.1.2 Mixed membership models · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe validations performed in the paper are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.
“Before proceeding, we caution that the validations performed here are only a subset of the evaluations any researcher would need to perform before using measures from unsupervised methods in their own work.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Premisenone givenIdentifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.
“Identifying the assumptions that will lead to a useful clustering of documents in a particular data set beforehand is difficult.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningSubstance-based validations are necessary to establish the utility of an unsupervised method.
“Regardless of the claims made, substance-based validations, as outlined in Section 6.4, are necessary to establish the utility of an unsupervised method.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningThe goal is revelation of substantively interesting information.
“The goal is revelation of substantively interesting information.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningWhile statistical structure allows inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.
“Further, while the statistical structure allows for the inclusion of problem-specific information, the model is still limited to a single family of probability distributions and therefore a specific set of assumptions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenThe topics must be labeled: it must be determined what each topic measures.
“As a prelude to validating the topic output, the topics must be labeled: it must be determined what each topic measures.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenStatistical methods for labeling topics share the intuition of identifying words that are highly predictive of documents belonging to a particular topic.
“While many have been proposed, they share the same intuition: identify words that are highly predictive of documents belonging to a particular topic.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenTo estimate a topic, topic models use the co-occurrence of words across documents.
“To estimate a topic, the models use the co-occurrence of words across documents.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThe similarity in structure across the three models demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.
“The similarity in structure across the three models in Fig. 2 demonstrates how statistical topic models can facilitate the production of substantively interesting and new models.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Premisenone givenAfter the fact, it is much easier to assess whether an organization of texts is useful within a particular context.
“After the fact, however, it is much easier to assess whether an organization of texts is useful within a particular context.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningThe post-fit validations necessary for unsupervised methods can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.
“The post-fit validations necessary can be extensive, but the methods are still useful because they suggest new, or at least understudied, ways to organize the data.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenSituating models in a data-generating process makes including additional structure straightforward.
“Situating the models in a data-generating process makes including additional structure straightforward.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Premisenone givenThe preprocessed texts represent a substantial simplification of the documents.
“The preprocessed texts represent a substantial simplification of the documents.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenK-means, like many other FAC algorithms, employs an approximate and iterative optimization method.
“K-means, like many other FAC algorithms, employs an approximate and iterative optimization method.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningTo make these inferences requires extensive validation of both the estimated topics and expressed priorities.
“To make these inferences, however, requires extensive validation of both the estimated topics and expressed priorities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningCAC methods provide a way to explore different assumptions.
“CAC methods provide a way to explore these different assumptions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenCAC provides an accelerated way to explore new collections of texts.
“Second, CAC provides an accelerated way to explore new collections of texts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Premisenone givenMany other distance metrics are possible, and each will lead to different clusterings.
“Many other distance metrics are possible, and each will lead to different clusterings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenCAC places substantial burden on users to identify clusterings that are useful.
“But CAC does place substantial burden on users to identify clusterings that are useful.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningCAC will only perform as well as the person conducting the exploration and interrogation of the space.
“But, it does imply that the method will only perform as well as the person conducting the exploration and interrogation of the space.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGrimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.
“To obtain the measures of expressed priorities, Grimmer (2012) applies a version of the expressed agenda model to over 64,000 Senate press releases issued from 2005 to 2007.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationnone givenThis type of credit claiming, bureaucratic agencies creating opportunities for legislators, is often missed from standard models of legislative speech.
“This is a prominent type of credit claiming-bureaucratic agencies creating opportunities for legislators-often missed from standard models of legislative speech.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSlapin and Proksch (2008) developed wordfish as a Poisson-IRT model.
“Slapin and Proksch (2008) develop their method, wordfish, as a Poisson-IRT model.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Slapin and Proksch 2008 source
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningWhen applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.
“Therefore when applying scaling algorithms, careful validation is needed to confirm that the intended space has been identified.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenWordscores is a special case of the dictionary methods presented in Section 5.1.
“Wordscores is a special case of the dictionary methods that we presented in Section 5.1.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningBy defining liberal and conservative using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.
“By defining “liberal” and “conservative” using only reference texts, Laver, Benoit, and Garry (2003) conflate ideological language with stylistic differences across authors and impose the ideological dominance assumption on the texts.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Laver, Benoit, and Garry 2003 source
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe expressed agenda model produces a measure of senators' expressed priorities: how senators divide their attention over the topics.
“The expressed agenda model also produces a measure of senators' expressed priorities: how senators divide their attention over the topics.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLaver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors, wordscores, rather than relying on manual coding or dictionary methods.
“Rather than rely on difficult to replicate and hard to validate manual coding or dictionary methods, Laver, Benoit, and Garry (2003) introduced a fully automated method for scaling political actors: wordscores.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Laver, Benoit, and Garry 2003 source
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningA method that could use text to place actors in a political space would facilitate testing some of the most important theories of politics.
“A method that could use this text to place actors in a political space would facilitate testing some of the most important theories of politics.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationcitationFacets of wordscores constrain the method and make it difficult to recommend for general use.
“But facets of wordscores constrain the method and make it difficult to recommend for general use (see Lowe 2008 for an extended critique).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Lowe 2008 source
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenNearly all political actors speak.
“But nearly all political actors speak.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningThe explicit adoption of a supervised learning approach might limit the influence of style substantially.
“The explicit adoption of a supervised learning approach might limit the influence of style substantially.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningImproving validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.
“Improving the validation of scales will help improve current models, which rely on the strong assumption of ideological dominance in speech.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationreasoningThe validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.
“The validation using the supervised method is possible only after the unsupervised method suggests a classification scheme.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
InterpretationreasoningThis validation does not obviate the need for unsupervised methods.
“We caution that this validation does not obviate the need for unsupervised methods.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenAny one method's performance will be context specific.
“We emphasize that any one method's performance will be context specific.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisereasoningEstimating locations using existing data is often difficult and sometimes impossible.
“Estimating locations using existing data is often difficult and sometimes impossible.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningEvery use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.
“The result is that every use of wordscores will depend strongly on the reference texts that are used, in part because of stylistic differences across authors and in part because the reference texts will discuss non-ideological content.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationBeauchamp (2011) shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.
“Beauchamp (2011), for instance, shows that results are significantly improved by removing technical language, which coincides more with party power than with ideology.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Beauchamp 2011
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S1 (asserted), S2 (checkable).
PremisecitationLeaders of Senate committees, committee chairs and ranking members, should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.
“Grimmer (2010) argues that the leaders of Senate committees-committee chairs and ranking members-should allocate more attention to issues that fall under the jurisdiction of their committee than other senators.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Grimmer 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningOne should not assume that wordfish output measures an ideological location without careful validation.
“But this does suggest that one should not assume that wordfish output measures an ideological location without careful validation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationnone givenThe estimated expressed priorities require rigorous validation, a demonstration that they measure the concept claimed.
“This too requires rigorous validation-a demonstration that the estimated priorities measure the concept claimed.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe explicit adoption of a supervised learning approach requires a substantial increase in effort and time, making its application unwieldy.
“Unfortunately, this also requires a substantial increase in effort and time, which makes it application unwieldy.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGrimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.
“Grimmer (2012) aggregates across categories to create an estimate of the number of press releases senators issue claiming credit for money in their state.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Grimmer 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
InterpretationreasoningApplying any one of the methods described without careful thought will likely lead to few answers and a great deal of frustration.
“Applying any one of the methods described here without careful thought will likely lead to few answers and a great deal of frustration.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningIf political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.
“If political scientists can effectively use large collections of texts in their inferences, then many substantively important questions are likely to be answered.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningFor unsupervised classification and scaling methods, validation requires validating that the measures produced correspond with the concepts claimed.
“For unsupervised classification and scaling methods, this requires validating that the measures produced correspond with the concepts claimed.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone givenBecause text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
Quote in the paperNot a substantial claim- Rests on an assumption: Text analysis methods are necessarily incorrect models of language.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.
Interpretationnone givenIf scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.
“If scholars recognize the limitations of statistical text models and demonstrate the validity of their measurements, automated methods will reach their promise and revolutionize fields of study within political science.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningFor supervised classification methods, validation requires demonstrating that the classification from machines replicates hand coding.
“For supervised classification methods, this requires demonstrating that the classification from machines replicates hand coding.”
Quote in the paperNot a substantial claim
The authors state this as an assumption.
Documents are a bag of words, where word order does not inform analyses.
“We will assume documents are a bag of words, where order does not inform our analyses.”
The authors state this as an assumption.
Ideology dominates the language used in the text.
“both methods rely heavily on the assumption that ideology dominates the language used in the text.”
The paper relies on this without support.
For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.
“For dictionary methods to work well, the scores attached to words must closely align with how the words are used in a particular context.”
The paper relies on this without support.
Measures from dictionaries are rarely validated.
“But measures from dictionaries are rarely validated.”
The paper relies on this without support.
Standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.
“Rather, standard practice in using dictionaries is to assume the measures created from a dictionary are correct and then apply them to the problem.”
The authors state this as an assumption.
Given a document's category, the words are generated independently.
“the model assumes that, given a document's category, the words are generated independently”
The authors state this as an assumption.
There is some unobserved function that describes the relationship between the words and the labels.
“Each supervised learning algorithm assumes that there is some (unobserved) function that describes the relationship between the words and the labels”
The authors state this as an assumption.
The conditional distributions are identical in the training and test sets.
“But if we assume that the conditional distributions are identical in the training and test sets, then we can substitute pðW test jC test Þ with pðW train jC train Þ.”
The authors state this as an assumption.
LDA assumes that each document is a mixture of topics.
“LDA assumes that each document is a mixture of topics.”
The authors state this as an assumption.
LDA assumes that a two-step process generates each word.
“LDA assumes that a two-step process generates each word.”
The authors state this as an assumption.
The expressed agenda model's key assumption is that each senator divides her attention over the set of topics.
“The key assumption is that each senator divides her attention over the set of topics: senators are at the mixed membership level in the model.”
The authors state this as an assumption.
Each document's proportions are drawn from a common Dirichlet prior.
“We will suppose that each document's proportions are drawn from a common Dirichlet prior, i $ Dirichlet ðÞ where represent the Dirichlet distribution's shape parameters.”
The authors state this as an assumption.
Actors' ideological leanings determine what is discussed in texts.
“Both supervised and unsupervised scaling methods rely on the strong assumption that actors' ideological leanings determine what is discussed in texts.”
The authors state this as an assumption.
Politicians reside in a low-dimensional political space.
“Politicians are assumed to reside in a low-dimensional political space, which is represented by the parameter i for politician i.”
The authors state this as an assumption.
A politician's or party's position in the political space affects the rate at which words are used in texts.
“A politician's (or party's) position in this space is assumed to affect the rate at which words are used in texts.”
The authors state this as an assumption.
Each word j from individual i is drawn from a Poisson distribution with rate ij.
“Specifically, Slapin and Proksch (2008) assume that each word j from individual i, W ij is drawn from a Poisson distribution with rate ij , W ij $ Poissonð ij Þ.”
The authors state this as an assumption.
The Poisson rate ij is modeled as a function of individual loquaciousness, word frequency, word discrimination, and politician position.
“ij is modeled as a function of individual i's loquaciousness ( i ), the frequency word j is used ( j ), the extent to which a word discriminates the underlying ideological space ( j ), and the politician's underlying position ( i ),”
The paper relies on this without support.
Text analysis methods are necessarily incorrect models of language.
“And because text analysis methods are necessarily incorrect models of language, the output always necessitates careful validation.”
The paper relies on this without support.
The Senate is now polarized.
“a necessity for any valid scaling in the now polarized Senate.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Topic models are rarely used as a mechanism for studying how individual words are used to convey different ideas.
“Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
From Embedding Regression: Models for Context-Specific Description and Inference (2023) · cites this paper as Grimmer and Stewart 2013
Section 6.1.2 explains that LDA assigns individual words to topics and describes applications that measure attention to topics, supporting the sentence’s description of how topic models work. It does not say that topic models are rarely used to study how individual words convey different ideas; that frequency claim remains unsupported. (LLM, conf=0.80)
Experimental — accuracy not yet measuredNatural language processing methods are already used in the field of political science to analyze political texts and legislation.
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [307]; these approaches could be promising for systematically studying climate change policies.”
From Tackling climate change with machine learning (2023) · cites this paper as [307]
Dictionary methods may signal something different when used in a new domain.
“they note that dictionary methods (such as the closed-vocabulary analyses discussed here) may signal something different when used in a new domain (for example ‘crude’ may be a negative word in student essays, but be neutral in energy industry reports: ‘crude oil’).”
From Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach (2013) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Naïve Bayes is said to assign documents to other documents
MinorThe description says the classifier assigns each test document to another document, whereas the paper defines its task as assigning documents to categories. The surrounding explanation indicates that "document" is an error here.
“Naı¨ve Bayes then assigns each document in the test set to the document with the highest probability.”
“The goal is to infer the probability that document i belongs to category k given word profile W i .”
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · see citing works
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,305 · see citing works
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
(1967) · Cited by 23,719 · see citing works
Neural networks for pattern recognition
Bishop, C. (1994) · Choice Reviews Online · doi:10.5860/choice.31-5500 · Cited by 18,537 · see citing works
Modern Applied Statistics with S
Venables, W., & Ripley, B. (2010) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 16,188 · see citing works
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448 · see citing works
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999) · ACM Computing Surveys · doi:10.1145/331499.331504 · Cited by 13,532 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · doi:10.1093/oso/9780198538493.001.0001 · Cited by 12,287 · see citing works
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278 · see citing works
Robust Locally Weighted Regression and Smoothing Scatterplots
Cleveland, W. S. (1979) · Journal of the American Statistical Association · doi:10.1080/01621459.1979.10481038 · Cited by 11,594 · see citing works
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288 · see citing works
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 2× · see citing works
“For more on the schemes—including how to assess coder agreement and practical guides to scheme creation—see Krippendorff (2004); Neuendorf (2002); Weber (1990) and the documentation available in the ReadMe software (Hopkins et al. 2010).”
MethodEnsemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408 · see citing works
On Spectral Clustering: Analysis and an algorithm
Ng, A. Y., Jordan, M. I., & Weiss, Y. (2001) · Cited by 8,159 · see citing works
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · see citing works
Weber, R. (1990) · doi:10.4135/9781412983488 · Cited by 7,750 · see citing works
Thumbs up?: sentiment classification using machine learning techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02 · doi:10.3115/1118693.1118704 · Cited by 7,024 · see citing works
Clustering by Passing Messages Between Data Points
Frey, B. J., & Dueck, D. (2007) · Science · doi:10.1126/science.1136800 · Cited by 6,733 · see citing works
When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10‐Ks
LOUGHRAN, T., & MCDONALD, B. (2011) · The Journal of Finance · doi:10.1111/j.1540-6261.2010.01625.x · Cited by 6,015 · see citing works
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · see citing works
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132 · see citing works
Congress, The Electoral Connection
Ross, M. J., & Mayhew, D. R. (1975) · The Western Political Quarterly · doi:10.2307/447379 · Cited by 4,094 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Probabilistic Latent Semantic Indexing
Hofmann, T. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130370 · Cited by 3,988 · see citing works
Probabilistic latent semantic indexing
Hofmann, T. (1999) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/312624.312649 · Cited by 3,975 · see citing works
Proceedings of the fifth Berkeley symposium on mathematical statistics and probability.
Kojima, K.-I. (1969) · Europe PMC (PubMed Central) · Cited by 3,648 · see citing works
The interaction of TIGIT with PVR and PVRL2 inhibits human NK cell cytotoxicity
Stanietsky, N., Simic, H., Arapovic, J., Toporik, A., Levy, O., Novik, A., Levine, Z., Beiman, M., Dassa, L., Achdout, H., et al. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0903474106 · Cited by 3,524 · see citing works
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229 · see citing works
Congress: The Electoral Connection, by David R. Mayhew
Page, B. I. (1975) · Political Science Quarterly · doi:10.2307/2148248 · Cited by 3,141 · see citing works
Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability; Vol. IV
Roy, H. L. L., LeCam, L. M., & Neyman, J. (1969) · Revue de l Institut International de Statistique / Review of the International Statistical Institute · doi:10.2307/1402306 · Cited by 2,878 · see citing works
Congress: A Political-Economic History of Roll Call Voting
Poole, K. T., & Rosenthal, H. L. (2000) · Medical Entomology and Zoology · Cited by 2,798 · see citing works
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,782 · see citing works
Advances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,701 · see citing works
Aldenderfer, M. S., & Blashfield, R. K. (1984) · Cited by 2,358 · see citing works
Linguistic Inquiry and Word Count (LIWC)
Chung, C. K., & Pennebaker, J. W. (2012) · IGI Global eBooks · doi:10.4018/978-1-60960-741-8.ch012 · Cited by 2,152 · see citing works
(2009) · Cited by 1,900 · see citing works
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852 · see citing works
The Political Economy of Benefits and Costs: A Neoclassical Approach to Distributive Politics
Weingast, B. R. V., Shepsle, K. A., & Johnsen, C. (1981) · Journal of Political Economy · doi:10.1086/260997 · Cited by 1,738 · see citing works
Variational inference for Dirichlet process mixtures
Jordan, M. I., & Jordan, M. I. (2006) · Bayesian Analysis · doi:10.1214/06-ba104 · Cited by 1,645 · see citing works
Home Style: House Members in Their Districts, by Richard F. Fenno, Jr.
Leuthold, D. A. (1979) · Political Science Quarterly · doi:10.2307/2150167 · Cited by 1,602 · see citing works
Measuring praise and criticism
Turney, P. D., & Littman, M. L. (2003) · ACM Transactions on Information Systems · doi:10.1145/944012.944013 · Cited by 1,515 · see citing works
van der Laan, M. J., Polley, E., & Hubbard, A. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1309 · Cited by 1,472 · see citing works
Extracting Policy Positions from Political Texts Using Words as Data
LAVER, M., BENOIT, K., & GARRY, J. (2003) · American Political Science Review · doi:10.1017/s0003055403000698 · Cited by 1,377 · see citing works
The general inquirer: A computer approach to content analysis.
Smith, M. S., Stone, P. J., & Dunphy, D. C. (1966) · Medical Entomology and Zoology · Cited by 1,165 · see citing works
The Statistical Analysis of Roll Call Data
CLINTON, J., JACKMAN, S., & RIVERS, D. (2004) · American Political Science Review · doi:10.1017/s0003055404001194 · Cited by 1,153 · see citing works
The General Inquirer: A Computer Approach to Content Analysis
Green, B. F., Stone, P. J., Dunphy, D. C., Smith, M. S., & Ogilvie, D. M. (1967) · American Educational Research Journal · doi:10.2307/1161774 · Cited by 1,013 · see citing works
On Relevance, Probabilistic Indexing and Information Retrieval
Maron, M. E., & Kuhns, J. L. (1960) · Journal of the ACM · doi:10.1145/321033.321035 · Cited by 908 · see citing works
Going negative : how political advertisements shrink and polarize the electorate
Ansolabehere, S. D., & Iyengar, S. (1997) · Cited by 826 · see citing works
Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · see citing works
A Scaling Model for Estimating Time‐Series Party Positions from Texts
Slapin, J. B., & Proksch, S. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00338.x · Cited by 684 · see citing works
How to analyze political attention with minimal assumptions and costs
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00427.x · Cited by 671 · see citing works
Scaling Policy Preferences from Coded Political Texts
Lowe, W., Benoit, K. R., Mikhaylov, S., & Laver, M. (2011) · Legislative Studies Quarterly · doi:10.1111/j.1939-9162.2010.00006.x · Cited by 654 · see citing works
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 636 · see citing works
Affective News: The Automated Coding of Sentiment in Political Texts
Young, L., & Soroka, S. (2012) · Political Communication · doi:10.1080/10584609.2012.671234 · Cited by 609 · cited 2× · see citing works
“Scholars have long recognized this, but the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research.”
BackgroundEstimating Policy Positions from Political Texts
Laver, M., & Garry, J. (2000) · American Journal of Political Science · doi:10.2307/2669268 · Cited by 574 · see citing works
Grimmer, J. (2009) · Political Analysis · doi:10.1093/pan/mpp034 · Cited by 552 · see citing works
Self-citationMonroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008) · Political Analysis · doi:10.1093/pan/mpn018 · Cited by 542 · see citing works
Mosteller, F., & Wallace, D. L. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500849 · Cited by 466 · see citing works
Blei, D. M. (2011) · doi:10.1145/2107736.2107741 · Cited by 434 · see citing works
Treating Words as Data with Error: Uncertainty in Text Statements of Policy Positions
Benoit, K., Laver, M., & Mikhaylov, S. (2009) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00383.x · Cited by 356 · see citing works
Beyond the Median: Voter Preferences, District Heterogeneity, and Political Representation
Gerber, E. R., & Lewis, J. B. (2004) · Journal of Political Economy · doi:10.1086/424737 · Cited by 324 · cited 1× · see citing works
“Other methods for scaling political actors have been developed (e.g., Gerber and Lewis 2004; Bonica 2011), but they rely on particular disclosure institutions that are often absent in other democracies.”
MethodMPs for Sale? Returns to Office in Postwar British Politics
EGGERS, A. C., & HAINMUELLER, J. (2009) · American Political Science Review · doi:10.1017/s0003055409990190 · Cited by 318 · see citing works
Coder Reliability and Misclassification in the Human Coding of Party Manifestos
Mikhaylov, S., Laver, M., & Benoit, K. R. (2012) · Political Analysis · doi:10.1093/pan/mpr047 · Cited by 250 · see citing works
Super Learning: An Application to the Prediction of HIV-1 Drug Resistance
Sinisi, S. E., Polley, E. C., Petersen, M. L., Rhee, S., & van der Laan, M. J. (2007) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1240 · Cited by 238 · see citing works
Grimmer, J. (2013) · American Journal of Political Science · doi:10.1111/ajps.12000 · Cited by 204 · see citing works
Self-citationMeasuring Praise and Criticism: Inference of Semantic Orientation from Association
Turney, P. D., & Littman, M. L. (2003) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0309034 · Cited by 201
Inference in an Authorship Problem
Mosteller, F. C., & Wallace, D. K. (1963) · Journal of the American Statistical Association · doi:10.2307/2283270 · Cited by 187 · see citing works
Delivering the Goods: Legislative Particularism in Different Electoral and Institutional Settings
Ashworth, S., & de Mesquita, E. B. (2005) · The Journal of Politics · doi:10.1111/j.1468-2508.2006.00378.x · Cited by 184 · see citing works
General purpose computer-assisted clustering and conceptualization
Grimmer, J., & King, G. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018067108 · Cited by 179 · see citing works
Self-citationSpeak Softly and Carry a Big Stick? Veterans in the Political Elite and the American Use of Force
Gelpi, C., & Feaver, P. D. (2002) · American Political Science Review · doi:10.1017/s000305540200045x · Cited by 179 · see citing works
Language and Ideology in Congress
Diermeier, D. A., Godbout, J., Yu, B., & Kaufmann, S. C. (2011) · British Journal of Political Science · doi:10.1017/s0007123411000160 · Cited by 169 · see citing works
Lowe, W. (2008) · Political Analysis · doi:10.1093/pan/mpn004 · Cited by 166 · see citing works
Armstrong, J. S. (1967) · The American Statistician · doi:10.1080/00031305.1967.10479849 · Cited by 165 · see citing works
Media Framing and the Dynamics of Racial Policy Preferences
Kellstedt, P. M. (2000) · American Journal of Political Science · doi:10.2307/2669308 · Cited by 159 · see citing works
Oracle inequalities for multi-fold cross validation
Vaart, A. W. V. D., Dudoit, S., & Laan, M. J. V. D. (2006) · Statistics & Decisions · doi:10.1524/stnd.2006.24.3.351 · Cited by 156 · see citing works
Computer-Assisted Topic Classification for Mixed-Methods Social Science Research
Hillard, D., Purpura, S., & Wilkerson, J. (2008) · Journal of Information Technology & Politics · doi:10.1080/19331680801975367 · Cited by 138 · see citing works
Media Framing and the Dynamics of Racial Policy Preferences
Kellstedt, P. M. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511615634.006 · Cited by 131 · see citing works
A Robust Transformation Procedure for Interpreting Political Text
Martin, L. W., & Vanberg, G. (2007) · Political Analysis · doi:10.1093/pan/mpm010 · Cited by 102 · see citing works
U.S. Treaty Making with American Indians: Institutional Change and Relative Power, 1784–1911
Spirling, A. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00558.x · Cited by 98 · see citing works
UK OC OK? Interpreting Optimal Classification Scores for the U.K. House of Commons
Spirling, A., & McLean, I. (2007) · Political Analysis · doi:10.1093/pan/mpl009 · Cited by 94 · see citing works
Do they work? Validating computerised word frequency estimates against policy series
Budge, I., & Pennings, P. (2006) · Electoral Studies · doi:10.1016/j.electstud.2006.04.002 · Cited by 76 · see citing works
Use of force and civil–military relations in Russia: an automated content analysis
Stewart, B. M., & Zhukov, Y. M. (2009) · Small Wars & Insurgencies · doi:10.1080/09592310902975455 · Cited by 69 · see citing works
Self-citationHouse Members' Communication Styles: Newsletters and Press Releases
Yiannakis, D. E. (1982) · The Journal of Politics · doi:10.2307/2130673 · Cited by 63 · see citing works
The Tone of Local Presidential News Coverage
Eshbaugh‐Soha, M. (2010) · Political Communication · doi:10.1080/10584600903502623 · Cited by 60 · see citing works
Budget Rhetoric in Presidential Campaigns from 1952 to 2000
Burden, B. C., & Sanberg, J. N. R. (2003) · Political Behavior · doi:10.1023/a:1023825212333 · Cited by 50 · see citing works
An Alternative Prior Process for Nonparametric Bayesian ClusteringPreprint
Wallach, H., Jensen, S. T., Dicker, L. H., et al. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0801.0461 · Cited by 44 · see citing works
Richards, D. (2000) · University of Michigan Press eBooks · doi:10.3998/mpub.16428 · Cited by 41 · see citing works
Twenty Years of the Kansas Event Data System Project
Schrodt, P. A. (2006) · Cited by 40 · see citing works
Shultziner, D. (2013) · Political Analysis · doi:10.1093/pan/mps035 · Cited by 35 · see citing works
Extracting Systematic Social Science Meaning from Text 1
Hopkins, D. J., & King, G. (2007) · Cited by 24 · see citing works
ReadMe: Software for Automated Content Analysis
King, G., Knowles, M., & Melendez, S. (2010) · Cited by 18 · see citing works
Inverse Regression for Analysis of Sentiment in TextPreprint
Taddy, M. A. (2010) · arXiv (Cornell University) · Cited by 12 · see citing works
Rejoinder: Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000079 · Cited by 10 · see citing works
Speak softly and carry a big stick
Joosen, E. P. M. (2010) · Data Archiving and Networked Services (DANS) · Cited by 9 · see citing works
Adler, E. S. (2011) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199559947.003.0021 · Cited by 7 · see citing works
RTextTools: A Supervised Learning Package for Text Classification
Boydstun, A. E., Collingwood, L., Grossman, E., et al. (2009) · SPIRE (Sciences Po) · Cited by 5 · see citing works
Brouard, S., Baumgartner, F. R., Wilkerson, J., et al. (2008) · Research Explorer (The University of Manchester) · Cited by 3 · see citing works
Multinomial Inverse Regression for Text AnalysisPreprint
Taddy, M. A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2098 · Cited by 1 · see citing works
Congress: The Electoral Connection
Mayhew, D. (1974) · Yale University Library · doi:10.60600/yu/b5awcm
Lowe, W. (2008) · doi:10.2139/ssrn.1095280
RTextTools: Automatic Text Classification via Supervised Learning
(2011) · CRAN: Contributed Packages · doi:10.32614/cran.package.rtexttools
Flagged first, then by citations
4 of the 109 references couldn’t be matched to a known paper.
Research works printed but not linked: 12
- Our review of clustering algorithms is necessarily limited. For example, we make no mention of the distinction between hierarchical and nonhierarchical clustering algorithms (Manning, Raghavan, and Schütze 2008). We avoid this distinction between hierarchical clustering algorithms and clustering algorithms that estimate several partitions. As a result, there is a direct mapping from the output of hierarchical clustering algorithms to the partitions we describe here. We also avoid the distinction between soft and hard clustering. Our experience with soft clustering is that these examples usually assign most of a document to essentially one category, resulting in essentially a hard clustering. We do consider mixed membership models below, another form of soft clustering.
- Hart, 2000, Diction 5.0: The text analysis program
- Greene, 2007, Econometric analysis
- Bonica, 2011, Estimating ideological positions of candidates and contributions from campaign finance records
- van der Vaart, 2006, Oracle inequalities for multifold cross validation, Statistics and Decisions, 24
- ReadMe estimates proportions, so the number of press releases in each category can be retrieved by multiplying by the total number of press releases.
- Berinsky, 2012, Using mechanical turk as a subject recruitment tool for experimental research, Political Analysis, 20
- Bradley, 1999, Affective Norms for English Words (ANEW): Stimuli, instruction, manual and affective ratings
- Jackman, 2006, Data from Web into R, The Political Methodologist, 14
- Fenno, 1978, Home style: House members in their districts
- Lodhi, 2002, Text classifications using string kernels, Journal of Machine Learning Research, 2
- Ng, 2001, Advances in neural information processing systems 14: Proceeding of the 2001 conference
Cited by
Sorted by citations
Design Principles for Industrie 4.0 Scenarios
Hermann, M., Pentek, T., & Otto, B. (2016) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2016.488 · Cited by 2,978 · see citing works
“[23] J. Grimmer, and B.M. Steward, “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts”, Political Analysis, 2013, pp. 1-31.”
BackgroundMethodInfluentialStructural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B., Tingley, D. H., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B. L., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 1,969 · see citing works
“…recent efforts in digitalization of text, researchers are rapidly reinventing the evidence base of the social sciences by introducing new measures of political and social behavior constructed from large text archives (Grimmer and Stewart 2013; Hopkins and King 2012; King 2009; Lazer et al. 2009).”
BackgroundSelf-citationPersonality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700 · see citing works
“For comprehensive surveys on text analyses across fields see Grimmer & Stewart [43], O’Connor, Bamman, & Smith [42], and Tausczik & Pennebaker [46].”
BackgroundResultInfluential<b>stm</b> : An <i>R</i> Package for Structural Topic Models
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019) · Journal of Statistical Software · doi:10.18637/jss.v091.i02 · Cited by 1,714 · see citing works
“These relationships can also play a key role in validating the topic model’s usefulness (Grim-mer 2010; Grimmer and Stewart 2013).”
BackgroundMethodSelf-citationGentzkow, M., Kelly, B. T., & Taddy, M. (2017) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,196 · see citing works
“See Evans and Aceves (2016) and Grimmer and Stewart (2013) for related surveys focused on text analysis in sociology and political science, respectively.”
BackgroundDiMaggio, P., Nag, M., & Blei, D. M. (2013) · Poetics · doi:10.1016/j.poetic.2013.08.004 · Cited by 1,152 · see citing works
“Like any clustering technique, the method should be employed as a heuristic tool in combination with additional information by a research team that includes subject-area experts (Grimmer and Stewart, 2011).”
BackgroundMethodInfluentialApplying LDA Topic Modeling in Communication Research: Toward a Valid and Reliable Methodology
Maier, D., Waldherr, A., Miltner, P., Wiedemann, G., Niekler, A., Keinert, A., Pfetsch, B., Heyer, G., Reber, U., Häußler, T., et al. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1430754 · Cited by 849 · see citing works
“Compared with other topic-clustering methods, a further advantage of LDA topic modeling is its mixed membership approach (Grimmer & Stewart, 2013, p. 18), i.e., one document can contain several topics, which is a useful assumption.”
BackgroundMethodAutomatic personality assessment through social media language.
Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., Ungar, L. H., & Seligman, M. E. P. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000020 · Cited by 802 · see citing works
“Incontrast, techniques from computational linguistics offer finer-grained, open-vocabulary methods for language analysis (e.g., Grimmer & Stewart, 2013; O’Connor, Bamman, & Smith, 2011; Schwartz et al., 2013b; Yarkoni, 2010).”
MethodThe double‐edged sword of recombination in breakthrough innovation
Kaplan, S., & Vakili, K. (2014) · Strategic Management Journal · doi:10.1002/smj.2294 · Cited by 770 · see citing works
A Model of Text for Experimentation in the Social Sciences
Roberts, M. E., Stewart, B., & Airoldi, E. M. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1141684 · Cited by 734 · see citing works
“An overview of manual validation procedures can be found in Grimmer and Stewart (2013).”
BackgroundMethodInfluentialSelf-citationComputational Grounded Theory: A Methodological Framework
Nelson, L. K. (2017) · Sociological Methods & Research · doi:10.1177/0049124117729703 · Cited by 702 · see citing works
“Given these multiple choices, there are many ways just one algorithm will group words into topics, and there are multiple algorithms available, producing hundreds of ways to reasonably group a single corpus (Grimmer and Stewart 2011).”
BackgroundAutomated Text Analysis for Consumer Research
Humphreys, A., & Wang, R. J.-H. (2018) · Journal of Consumer Research · doi:10.1093/jcr/ucx104 · Cited by 636 · see citing works
Topic Modeling in Management Research: Rendering New Theory from Textual Data
Hannigan, T. R., Haans, R. F. J., Vakili, K., Tchalian, H., Glaser, V., Wang, M. S., Kaplan, S., & Jennings, P. D. (2019) · Academy of Management Annals · doi:10.5465/annals.2017.0099 · Cited by 625 · see citing works
“In the current emergent phase of its employment, scholars are still learning the best ways to reveal constructs and develop theory (Evans & Aceves, 2016; Grimmer & Stewart, 2013)—which implies a need for deeper insights into how topic modeling can inform new theories.”
BackgroundBeyond prediction: using big data for policy problems
Athey, S. (2017) · Science · doi:10.1126/science.aal4321 · Cited by 575 · see citing works
Stier, S., Bleier, A., Lietz, H., & Strohmaier, M. (2018) · Studying Politics Across Media · doi:10.1080/10584609.2017.1334728 · Cited by 544 · see citing works
“Recent advances in quantitative text analysis (Grimmer & Stewart, 2013) enable us to perform text analysis at a larger scale.”
BackgroundMethodInfluentialBarberá, P., Casas, A., Nagler, J., Egan, P. J., Bonneau, R., Jost, J. T., & Tucker, J. A. (2019) · American Political Science Review · doi:10.1017/s0003055419000352 · Cited by 542 · see citing works
The AfD: Finally a Successful Right-Wing Populist Eurosceptic Party for Germany?
Arzheimer, K. (2015) · West European Politics · doi:10.1080/01402382.2015.1004230 · Cited by 530 · see citing works
“While the precise meaning of ‘left’ and ‘right’ may vary across time and space (Huber and Inglehart 1995), left–right placement usually reflects the perceived distance between voters and parties as well as valuebased preferences (Knutsen 1997).”
BackgroundGuriev, S., & Treisman, D. (2019) · Journal of Economic Perspectives · doi:10.1257/jep.33.4.100 · Cited by 560 · see citing works
Quantitative analysis of large amounts of journalistic texts using topic modelling
Jacobi, C., van Atteveldt, W., & Welbers, K. (2015) · Digital Journalism · doi:10.1080/21670811.2015.1093271 · Cited by 532 · see citing works
“For a general overview of the available methods and pitfalls for topic models, see Grimmer and Stewart (2013).”
MethodAgenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting
Wasow, O. (2020) · American Political Science Review · doi:10.1017/s000305542000009x · Cited by 517 · see citing works
“The text of the headlines was processed according to standard procedures (Grimmer and Stewart 2013).”
MethodSimultaneously Discovering and Quantifying Risk Types from Textual Risk Disclosures
Bao, Y., & Datta, A. (2014) · Management Science · doi:10.1287/mnsc.2014.1930 · Cited by 491 · see citing works
“They are valuable since they could identify organizations of texts that are theoretically useful but perhaps understudied or previously unknown (Grimmer and Stewart 2013).”
BackgroundMethodInfluentialPsychological Language on Twitter Predicts County-Level Heart Disease Mortality
Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G., Labarthe, D. R., Merchant, R. M., Jha, S., Agrawal, M., Dziurzynski, L., Sap, M., et al. (2015) · Psychological Science · doi:10.1177/0956797614557867 · Cited by 487 · see citing works
“Because words can have multiple senses, act as multiple parts of speech, and be used in the context of irony or negation, it is important to gauge empirically how well such lists of words measure what is intended (Grimmer & Stewart, 2013).”
BackgroundIntroduction—Topic models: What they are and why they matter
Mohr, J. W., & Bogdanov, P. (2013) · Poetics · doi:10.1016/j.poetic.2013.10.001 · Cited by 489 · see citing works
Computer-Assisted Text Analysis for Comparative Politics
Lucas, C., Nielsen, R. A., Roberts, M. E., Stewart, B., Storer, A., & Tingley, D. H. (2015) · Political Analysis · doi:10.1093/pan/mpu019 · Cited by 487 · see citing works
“Unlike the issues involved in multilingual text processing, these methods have been well developed elsewhere (e.g., Grimmer and Stewart 2013).”
MethodSelf-citationSmart literature review: a practical topic modelling approach to exploratory literature review
Asmussen, C. B., & Møller, C. (2019) · Journal Of Big Data · doi:10.1186/s40537-019-0255-7 · Cited by 467 · see citing works
Issues and Best Practices in Content Analysis
Lacy, S. R., Watson, B. R., Riffe, D., & Lovejoy, J. (2015) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699015607338 · Cited by 478 · see citing works
“…the algorithmic coder can be validated (e.g., Conway, 2006), the algorithmic coder is capable of analyzing texts at a level of granularity—i.e., counts of individual words—that humans cannot reliably replicate, particularly over a large number of “big data” study units (Grimmer & Stewart, 2013).”
BackgroundInfluentialThe SAGE Handbook of Online Research Methods
Fielding, N., Lee, R. M., & Blank, G. (2016) · SAGE Publications Ltd eBooks · doi:10.4135/9781473957992 · Cited by 435 · see citing works
Denny, M. J., & Spirling, A. (2017) · Political Analysis · doi:10.1017/pan.2017.44 · Cited by 502 · see citing works
“A commonly used rule of thumb is to discard terms that appear in less than 0.5-1% of documents (Grimmer, 2010; Yano, Smith and Wilkerson, 2012; Grimmer and Stewart, 2013), however, there has been no systematic study of the effects this preprocessing choice has on downstream analyses.”
BackgroundMethodInfluentialvan Atteveldt, W., van der Velden, M. A. C. G., & Boukes, M. (2021) · Communication Methods and Measures · doi:10.1080/19312458.2020.1869198 · Cited by 420 · see citing works
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation
Syed, S., & Spruit, M. R. (2017) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2017.61 · Cited by 427 · see citing works
“Topic modeling approaches have proved to be very helpful in elucidating the key ideas within a set of documents [8]–[10], and they do so with greater speed and a quantitative rigor that would otherwise be possible only through a traditional narrative review [9].”
BackgroundChen, X., Zou, D., Cheng, G., & Xie, H. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103855 · Cited by 406 · see citing works
The Populist Style in American Politics: Presidential Campaign Discourse, 1952–1996
Bonikowski, B., & Gidron, N. (2015) · Social Forces · doi:10.1093/sf/sov120 · Cited by 350 · see citing works
“5 As with any automated text-analysis method, validation is an essential step in the analysis (Grimmer and Stewart 2013).”
MethodSpontaneous Collective Action: Peripheral Mobilization During the Arab Spring
Steinert-Threlkeld, Z. C. (2017) · American Political Science Review · doi:10.1017/s0003055416000769 · Cited by 229 · see citing works
Machine Translation: Mining Text for Social Theory
Evans, J. A., & Aceves, P. (2016) · Annual Review of Sociology · doi:10.1146/annurev-soc-081715-074206 · Cited by 335 · see citing works
Tixier, A. J. ‐., Hallowell, M. R., Rajagopalan, B., & Bowman, D. (2015) · Automation in Construction · doi:10.1016/j.autcon.2015.11.001 · Cited by 338 · see citing works
Abductive Coding: Theory Building and Qualitative (Re)Analysis
Vila‐Henninger, L. A., Dupuy, C., Ingelgom, V. V., Caprioli, M., Teuber, F., Pennetreau, D., Bussi, M., & Gall, C. L. (2022) · Sociological Methods & Research · doi:10.1177/00491241211067508 · Cited by 338 · see citing works
“For instance, one might consider training a supervised machine learning algorithm based on an established abductive code-book and evaluate its validity via the application of a “ test set ” (Grimmer and Stewart 2013).”
MethodPierre Bourdieu: theorizing the digital
Ignatow, G., & Robinson, L. (2017) · Information Communication & Society · doi:10.1080/1369118x.2017.1301519 · Cited by 331 · see citing works
“Human coding can be partially automated through the application of supervised learning techniques (Grimmer & Stewart, 2013, pp. 9 – 10).”
MethodBoyd, R. L., & Schwartz, H. A. (2021) · Journal of Language and Social Psychology · doi:10.1177/0261927x20967028 · Cited by 317 · see citing works
“For example, political science papers describing computational text analysis (Grimmer & Stewart, 2013); psychology papers describing machine learning (Stachl et al., 2020), and machine learning papers describing social sciences (O’Connor et al., 2011).”
BackgroundBonikowski, B. (2016) · Annual Review of Sociology · doi:10.1146/annurev-soc-081715-074412 · Cited by 327 · see citing works
Directions in abusive language training data, a systematic review: Garbage in, garbage out
Vidgen, B., & Derczynski, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243300 · Cited by 335 · see citing works
“Traditional qualitative methods cannot scale to handle the hundreds of millions of posts which appear on each major social media platform every day, and introduce various inconsistencies and biases [8].”
BackgroundFrom Newsworthiness to Shareworthiness
Trilling, D., Tolochko, P., & Burscher, B. (2016) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699016654682 · Cited by 319 · see citing works
“SML is a technique in which a computer learns from a set of humancoded training documents to predict content-analytical variables in texts automatically (e.g., Grimmer & Stewart, 2013; Russell & Norvig, 2002).”
MethodCrowd-sourced Text Analysis: Reproducible and Agile Production of Political Data
Benoit, K. R., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016) · American Political Science Review · doi:10.1017/s0003055416000058 · Cited by 296 · see citing works
“This is related to, but quite distinct from, spectacular recent advances in automated text analysis that in theory scale up to unlimited volumes of political text (Grimmer and Stewart 2013).”
ResultLarge-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300 · see citing works
“Validation is a critical component of every text as data project (Saldana 2009; Grimmer and Stewart 2013).”
BackgroundLiu, Q., Zheng, Z., Zheng, J., Chen, Q., Liu, G. K., Chen, S., Chu, B., Zhu, H., Akinwunmi, B. O., Huang, J. V., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/19118 · Cited by 298 · see citing works
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295 · see citing works
Welbers, K., van Atteveldt, W., & Benoit, K. R. (2017) · Communication Methods and Measures · doi:10.1080/19312458.2017.1387238 · Cited by 291 · see citing works
“Although this drastically simplifies text content, research and many real-world applications show that word frequencies alone contain sufficient information for many types of analysis (Grimmer & Stewart, 2013).”
BackgroundMethodInfluentialText-mining the signals of climate change doubt
Boussalis, C., & Coan, T. (2015) · Global Environmental Change · doi:10.1016/j.gloenvcha.2015.12.001 · Cited by 266 · see citing works
“Grimmer and Stewart (2013), in a review of the text analysis litera378 ture in political science, argue emphatically for the need to “[v]alidate, validate, 379 validate,” stating “that what should be avoided, then, is the blind use of any 380 method without a validation step” (pg.”
BackgroundKilling in the Slums: Social Order, Criminal Governance, and Police Violence in Rio de Janeiro
Magaloni, B., Vivanco, E. F., & Melo, V. (2020) · American Political Science Review · doi:10.1017/s0003055419000856 · Cited by 256 · see citing works
“Following Grimmer and Stewart (2013), we rely on a rule-based method to classify each call into our typology of criminal behavior.”
MethodA network approach to topic models
Gerlach, M., Peixoto, T. P., & Altmann, E. G. (2018) · Science Advances · doi:10.1126/sciadv.aaq1360 · Cited by 245 · see citing works
“The latter established itself as the state-of-the-art method in topic modeling and has been widely used not only for recommendation and classification (7) but also for bibliometrical (8), psychological (9), and political (10) analysis.”
MethodCao, J., Ho, M. S., Ma, R., & Teng, F. (2021) · Journal of Public Economics · doi:10.1016/j.jpubeco.2021.104470 · Cited by 244 · see citing works
An Investigation of Brand-Related User-Generated Content on Twitter
Liu, X., Burns, A. C., & Hou, Y. (2017) · Journal of Advertising · doi:10.1080/00913367.2017.1297273 · Cited by 244 · see citing works
“Grimmer and Stewart (2013) warn researchers about the threats to validity in automatic text analysis and point out that quantitative text analysis should be used with convincing validation methods.”
BackgroundVries, E. D., Schoonvelde, M., & Schumacher, G. (2018) · Political Analysis · doi:10.1017/pan.2018.26 · Cited by 245 · see citing works
Nelson, L. K., Burk, D., Knudsen, M., & McCall, L. (2018) · Sociological Methods & Research · doi:10.1177/0049124118769114 · Cited by 239 · see citing works
Adapting computational text analysis to social science (and vice versa)
DiMaggio, P. (2015) · Big Data & Society · doi:10.1177/2053951715602908 · Cited by 231 · see citing works
“2 There have been promising developments in this respect (e.g. Grimmer and Stewart, 2013) and room for more progress.”
BackgroundComparing grounded theory and topic modeling: Extreme divergence or unlikely convergence?
Baumer, E. P. S., Mimno, D., Guha, S., Quan, E., & Gay, G. K. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23786 · Cited by 232 · see citing works
“Much of it centers around one method, statistical topic modeling (Grimmer & Stewart, 2013; Jockers, 2013; Mohr & Bogdanov, 2013; Rhody, 2012), which can be traced to work in the early 1990s on latent semantic analysis (Deerwester, Dumais, Furnas, Landauer, & Harshman, 1990).”
BackgroundMethodNews Frame Analysis: An Inductive Mixed-method Computational Approach
Walter, D., & Ophir, Y. (2019) · Communication Methods and Measures · doi:10.1080/19312458.2019.1639145 · Cited by 241 · see citing works
“In topic models, every word in the corpus has a probability of appearing in each of the topics, and every document is composed as a mixture of all topics (Grimmer & Stewart, 2013).”
BackgroundGerring, J. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-092415-024158 · Cited by 330 · see citing works
Gaining insights from social media language: Methodologies and challenges.
Kern, M. L., Park, G., Eichstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L. K., & Ungar, L. H. (2016) · Psychological Methods · doi:10.1037/met0000091 · Cited by 221 · see citing works
“In this last section, we highlight key issues related to processing and analyzing data, interpreting results, and ethical considerations (see also Grimmer & Stewart, 2013; Iliev, Dehghani, & Sagi, 2014; Shah, Cappella, & Neuman, 2015; Yarkoni, 2012 for methodologies and discussions of issues).”
BackgroundMore than Bags of Words: Sentiment Analysis with Word Embeddings
Rudkowsky, E., Haselmayer, M., Wastian, M., Jenny, M., Emrich, Š., & Sedlmair, M. (2018) · Communication Methods and Measures · doi:10.1080/19312458.2018.1455817 · Cited by 222 · see citing works
“From these results we conclude that the word embeddings approach offers a lot of potential for sentiment analysis and, more generally, automated text analysis in the social sciences (Boumans & Trilling, 2016; Grimmer & Stewart, 2013; Lowe & Benoit, 2013; Lucas et al., 2015; Wilkerson & Casas, 2017).”
BackgroundContent analysis of e-petitions with topic modeling: How to train and evaluate LDA models?
Hagen, L. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.05.006 · Cited by 233 · see citing works
Machine learning in agricultural and applied economics
Storm, H., Baylis, K., & Heckelei, T. (2019) · European Review of Agricultural Economics · doi:10.1093/erae/jbz033 · Cited by 227 · see citing works
Topic Modeling and Text Analysis for Qualitative Policy Research
Isoaho, K., Gritsenko, D., & Eetu Mäkelä, A. (2019) · Policy Studies Journal · doi:10.1111/psj.12343 · Cited by 231 · see citing works
“In the words of Grimmer and Stewart (2013, p. 5), which are often cited but deserve to be repeated: (T)he output of the models may be misleading or simply wrong.”
InfluentialBoumans, J. W., & Trilling, D. (2015) · Digital Journalism · doi:10.1080/21670811.2015.1096598 · Cited by 212 · see citing works
“Thus, rather than replacing humans, it is more correct to view computers as amplifying human abilities (Grimmer and Stewart 2013).”
BackgroundInfluentialSelective responsiveness: Online public demands and government responsiveness in authoritarian China
Su, Z., & Meng, T. (2016) · Social Science Research · doi:10.1016/j.ssresearch.2016.04.017 · Cited by 207 · see citing works
“In the digital era, the combination of massive text datawith automated text analysis has become one ofmajor research strategies to explore big data in political science (Grimmer and Stewart, 2013; Hopkins and King, 2010).”
BackgroundMethodHopp, F. R., Fisher, J., Cornell, D., Huskey, R., & Weber, R. (2020) · Behavior Research Methods · doi:10.3758/s13428-020-01433-0 · Cited by 230 · see citing works
The structural topic model and applied social sciencePreprint
Roberts, M. E., Stewart, B., Tingley, D. H., & Airoldi, E. M. (n.d.) · RePEc: Research Papers in Economics · Cited by 231 · see citing works
Self-citationSentiment analysis of political communication: combining a dictionary approach with crowdcoding
Haselmayer, M., & Jenny, M. (2016) · Quality & Quantity: International Journal of Methodology · doi:10.1007/s11135-016-0412-4 · Cited by 193 · see citing works
“…Monroe et al. 2008) and cross-validation (e.g., Laver et al. 2003; Slapin and Proksch 2008) are popular standards used to evaluate results from automated text analyses, but the gold standard is a comparison with results from human coding (Grimmer and Stewart 2013; Lowe and Benoit 2013, p. 13).”
BackgroundMethodInfluentialO’Grady, T. (2018) · Comparative Political Studies · doi:10.1177/0010414018784065 · Cited by 197 · see citing works
“There are no formal statistical tests of how well a particular piece of text analysis has performed (Grimmer & Stewart, 2013).”
BackgroundText mining for information systems researchers: An annotated topic modeling tutorial
University of Liechtenstein, Debortoli, S., Müller, O., IT University of Copenhagen, Junglas, I., Florida State University, vom Brocke, J., & University of Liechtenstein (2016) · Communications of the Association for Information Systems · doi:10.17705/1cais.03907 · Cited by 199 · see citing works
“This, for example, allows topics to be ranked by prevalence, to compare their prevalence for specific sub-groups, or to track the evolution of topics over time (see Grimmer & Stewart (2013) for examples).”
BackgroundWe Are All Social Scientists Now: How Big Data, Machine Learning, and Causal Inference Work Together
Grimmer, J. (2014) · PS Political Science & Politics · doi:10.1017/s1049096514001784 · Cited by 199 · see citing works
Self-citationSetting the Agenda or Responding to Voters? Political Parties, Voters and Issue Attention
Klüver, H., & Sagarzazu, I. (2015) · West European Politics · doi:10.1080/01402382.2015.1101295 · Cited by 201 · see citing works
“Similar to other works in the discipline (Grimmer and Stewart 2013), we performed a two-stage process to determine the number of issues.”
MethodResultGovernment Responsiveness in the European Union: Evidence From Council Voting
Hagemann, S., HOBOLT, S. B., & Wratil, C. (2016) · Comparative Political Studies · doi:10.1177/0010414015621077 · Cited by 196 · see citing works
“The word generating process within each document is then modeled by first drawing the topic and then, conditional on the topic, the respective word from a multinomial distribution (see Grimmer & Stewart, 2013).”
MethodA Review of Best Practice Recommendations for Text Analysis in R (and a User-Friendly App)
Banks, G. C., Woznyj, H. M., Wesslen, R. S., & Ross, R. L. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-017-9528-3 · Cited by 194 · see citing works
“Other new techniques like semantic coherence attempt to measure word consistency within topics as a proxy for measuring interpretability (Grimmer & Stewart, 2013).”
MethodIt doesn’t hurt to ask: Question-asking increases liking.
Huang, K., Yeomans, M., Brooks, A. W., Minson, J. A., & Francesca Gino, B. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000097 · Cited by 194 · see citing works
Aslanidis, P. (2017) · Quality & Quantity · doi:10.1007/s11135-017-0517-4 · Cited by 192 · see citing works
Topic models meet discourse analysis: a quantitative tool for a qualitative approach
Jacobs, T., & Tschötschel, R. (2019) · International Journal of Social Research Methodology · doi:10.1080/13645579.2019.1576317 · Cited by 187 · see citing works
“Automated text analysis tools always contain an implicit and necessarily imperfect model of how language and the generation of meaning through language work (Grimmer & Stewart, 2013, pp. 3 – 4).”
BackgroundMethodLexical shifts, substantive changes, and continuity in State of the Union discourse, 1790–2014
Rule, A., Cointet, J., & Bearman, P. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1512221112 · Cited by 188 · see citing works
“Such methods thus assume a specific substantive context (13)—for example, a dictionary for political discourse would not capture the meaning of the same terms used in everyday speech.”
Ash, E., & Hansen, S. (2023) · Annual Review of Economics · doi:10.1146/annurev-economics-082222-074352 · Cited by 197 · see citing works
“The standard approach has been extensively discussed elsewhere (e.g., Manning et al. 2008, Grimmer & Stewart 2013, Denny & Spirling 2018, Gentzkow et al. 2019a).”
Tackling climate change with machine learning
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., et al. (2023) · ACM Computing Surveys · doi:10.1145/3485128 · Cited by 336 · see citing works
“In particular, natural language processing methods are already used in the field of political science to analyze political texts and legislation [759]; these approaches could be promising for systematically studying climate change policies.”
BackgroundThree Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda
Baden, C., Pipal, C., Schoonvelde, M., & van der Velden, M. A. C. G. (2021) · Communication Methods and Measures · doi:10.1080/19312458.2021.2015574 · Cited by 183 · see citing works
“…absence of operational considerations in the development and documentation of CTAM, social scientists trying to make an informed choice about their use of computational methods frequently fail to find relevant guidance in the existing methodological literature (Grimmer & Stewart, 2013).”
BackgroundLi, J., Li, G., Liu, M., et al. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2020.02.002 · Cited by 178 · see citing works
Measuring Political Positions from Legislative Speech
Lauderdale, B. E., & Herzog, A. (2016) · Political Analysis · doi:10.1093/pan/mpw017 · Cited by 173 · see citing works
Detecting weak and strong Islamophobic hate speech on social media
Vidgen, B., & Yasseri, T. (2019) · Journal of Information Technology & Politics · doi:10.1080/19331681.2019.1702607 · Cited by 173 · see citing works
Stecula, D. A., & Merkley, E. (2019) · Frontiers in Communication · doi:10.3389/fcomm.2019.00006 · Cited by 185 · see citing works
“These techniques are increasingly used by scholars in the social sciences to study news content and political discourse more broadly (Young and Soroka, 2012; Grimmer and Stewart, 2013; Lacy et al., 2015).”
BackgroundWord Embeddings for the Analysis of Ideological Placement in Parliamentary Corpora
Rheault, L., & Cochrane, C. (2019) · Political Analysis · doi:10.1017/pan.2019.26 · Cited by 164 · see citing works
“Both parties are located next to each 17 Grimmer and Stewart (2013) have previously illustrated a limitation to WordFish when applied to individual legislators in Congress, a case for which the model does not perform well at capturing the expected ideological distance expected betweenmembers of…”
BackgroundDaenekindt, S., & Huisman, J. (2020) · Higher Education · doi:10.1007/s10734-020-00500-x · Cited by 169 · see citing works
“There is no ‘right’ number of topics, and the number of topics should be chosen based on interpretability and analytic utility with regard to the research question (DiMaggio et al. 2013; Grimmer and Stewart 2013; Roberts et al. 2014b).”
BackgroundMethodInfluentialLaurer, M., van Atteveldt, W., Casas, A., & Welbers, K. (2023) · Political Analysis · doi:10.1017/pan.2023.20 · Cited by 169 · see citing works
“…reason for the greater popularity of unsupervised approaches in the social sciences. unsupervised machine learning; or text classification with prior categories with dictionaries or supervised machine learning (Benoit 2020; Chatsiou and Mikhaylov 2020; Grimmer and Stewart 2013; Lucas et al .”
Understanding cultural differences and extreme attitudes in the 2021 OECD Trust Survey
Smid, S. (2023) · OECD working papers on public governance · doi:10.1787/ef25d883-en · Cited by 166 · see citing works
Policy Coordination and Integration: A Research Agenda
Trein, P., Biesbroek, R. G., Bolognesi, T., et al. (2020) · Public Administration Review · doi:10.1111/puar.13180 · Cited by 166 · see citing works
Devinney, T. M., & Hartwell, C. A. (2020) · Global Strategy Journal · doi:10.1002/gsj.1373 · Cited by 167 · see citing works
Data-Driven Content Analysis of Social Media
Schwartz, H. A., & Ungar, L. (2015) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716215569197 · Cited by 166 · see citing works
Grover, P., Kar, A. K., & Ilavarasan, P. V. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.01.009 · Cited by 163 · see citing works
Research trends in text mining: Semantic network and main path analysis of selected journals
Jung, H., & Lee, B. G. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113851 · Cited by 163 · see citing works
Automated Text Classification of News Articles: A Practical Guide
Barberá, P., Boydstun, A., Linn, S., McMahon, R., & Nagler, J. (2020) · Political Analysis · doi:10.1017/pan.2020.8 · Cited by 156 · see citing works
“Whereas researchers were once limited to analyses based on text that was read and hand-coded by humans, machine coding by dictionaries and supervised machine learning tools are now the norm (Grimmer & Stewart 2013).”
BackgroundWhat's in a buzzword? A systematic review of the state of populism research in political science
Hunger, S., & Paxton, F. (2021) · Political Science Research and Methods · doi:10.1017/psrm.2021.44 · Cited by 161 · see citing works
Using Sentiment Analysis to Monitor Electoral Campaigns
Cerón, A., Curini, L., & Iacus, S. M. (2014) · Social Science Computer Review · doi:10.1177/0894439314521983 · Cited by 158 · see citing works
“The traditional approach to sentiment analysis is in fact based on the use of ontological dictionaries: This means that a text is assigned to a specific opinion category if some predetermined words or expressions appear (or do not) in the text (see Grimmer & Stewart, 2013, on this point).”
MethodInfluentialMachine learning, artificial neural networks and social research
Franco, G. D., & Santurro, M. (2020) · Quality & Quantity · doi:10.1007/s11135-020-01037-y · Cited by 158 · see citing works
Tracing the Flow of Policy Ideas in Legislatures: A Text Reuse Approach
Wilkerson, J., Smith, D. A., & Stramp, N. (2015) · American Journal of Political Science · doi:10.1111/ajps.12175 · Cited by 154 · see citing works
Pitch Perfect: Vocal Pitch and the Emotional Intensity of Congressional Speech
Dietrich, B. J., HAYES, M., & O’Brien, D. Z. (2019) · American Political Science Review · doi:10.1017/s0003055419000467 · Cited by 154 · see citing works
Michalopoulos, S., & Xue, M. M. (2021) · The Quarterly Journal of Economics · doi:10.1093/qje/qjab003 · Cited by 165 · see citing works
- Authors
- Grimmer, J., & Stewart, B. M.
- Affiliations
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Grimmer, Justin
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2013
- Volume
- 21
- Issue
- 3
- Pages
- 267-297
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2095655043
- Cited by
- 3,297
- References
- 109
- Citations by year
- 2026: 437 · 2025: 340 · 2024: 243 · 2023: 308 · 2022: 330 · 2021: 329 · 2020: 305 · 2019: 276 · 2018: 239 · 2017: 168 · 2016: 128 · 2015: 103 · 2014: 55 · 2013: 26 · 2012: 8
- FWCI
- 537.75
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Analysis (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- National Science Foundation
- University of Cambridge
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Keywords
- computational text analysis, political texts, model validation, close reading, large document collections, political discourse, methodological critique
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
