CoChat
Sign Up
Cited by 1,714Computational and Text Analysis MethodsTopic ModelingBayesian Methods and Mixture Models

stm : An R Package for Structural Topic Models

Open access
Roberts, Margaret E., Stewart, Brandon M., Tingley, Dustin·2019·Journal of Statistical Software
DOI 10.18637/jss.v091.i02
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper demonstrates how to use the R package stm for structural topic modeling. The structural topic model allows researchers to flexibly estimate a topic model that includes document-level metadata. Estimation is accomplished through a fast variational approximation. The stm package provides many useful features, including rich ways to explore topics, estimate uncertainty, and visualize quantities of interest.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents the R package `stm` for structural topic modeling, which lets researchers incorporate document-level metadata into topic models through topical prevalence and topical content covariates. It demonstrates the package workflow and reports benchmark results showing faster estimation and better held-out log-likelihood than the comparable `topicmodels` CTM implementation.

Key findings

  • The STM’s key innovation is incorporating arbitrary document metadata into topic modeling; metadata can affect topical prevalence, topical content, both, or neither, and with no covariates the model reduces to a fast implementation of the correlated topic model.
  • The `stm` package supports an end-to-end workflow: ingest, prepare, estimate, evaluate, understand, and visualize, with functions such as `textProcessor`, `prepDocuments`, `stm`, `selectModel`, `searchK`, `estimateEffect`, `labelTopics`, `findThoughts`, and plotting methods.
  • In a benchmark using 5,000 randomly sampled poliblog documents with K = 20, the paper reports: STM default 21 iterations, 0.7 min total, 1.9 sec/iteration, held-out log-likelihood −6.900; CTM 34 iterations, 96.7 min, 170.7 sec/iteration, −6.935; STM alternate convergence 6 iterations, 0.3 min, 2.7 sec/iteration, −6.905; CTM alternate 15 iterations, 6.1 min, 24.3 sec/iteration, −7.040.
  • A random-initialized STM run is also reported at −6.92 held-out log-likelihood, 7.2 minutes, 280 iterations, and less than 1.5 seconds per iteration.

Methods

  • This is a software/design paper for the R package `stm`; the STM is a generative model of word counts where document-topic proportions come from a logistic-normal generalized linear model based on prevalence covariates, and topic-word distributions can include content covariate deviations.
  • Estimation uses a partially collapsed variational expectation-maximization algorithm with a Laplace approximation; the default initialization is spectral, with alternatives including LDA via collapsed Gibbs, random, and custom.
  • Examples use the CMU 2008 Political Blog Corpus, estimating a 20-topic STM with prevalence `~ rating + s(day)` and content `~ rating`; the benchmark uses 5,000 randomly sampled documents (`poliblog5k`), K = 20, and held-out log-likelihood with 10% of documents having half their words removed, compared against the `topicmodels` CTM.

Limitations

  • The paper states that STM assumes a fixed user-specified number of topics and that there is no “right” answer for K; spectral initialization is deterministic but may not produce exactly the same results across machines due to numerical precision; the Lee–Mimno K = 0 selection procedure has no particular statistical guarantees; content covariates are limited to a single non-continuous variable and computational cost rises quickly with the number of groups; `ngroups` trades memory for speed and may converge to a different solution; interactions can be turned off when there is insufficient data.
  • Evident from the text: the paper’s examples do not run the models but load a pre-estimated workspace, and the benchmark uses K = 20 chosen arbitrarily.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18637/jss.v091.i02

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 33 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The stm implementation of the correlated topic model demonstrates superior performance compared to the CTM.

    “The results clearly demonstrate the superior performance of the stm implementation of the correlated topic model.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    Through a series of design decisions, the stm package has been able to speed up the code considerably.

    “Nevertheless, through a series of design decisions which we highlight next we have been able to speed up the code considerably.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Bayesian inference for the structural topic model is challenging and computationally intensive due to the non-conjugate model structure.
  • Findingown data

    Comparing comparable convergence thresholds, the stm is able to run completely to convergence before the CTM has made a single pass through the data.

    “In fact, comparing comparable convergence thresholds the stm is able to run completely to convergence before the CTM has made a single pass through the data.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    The structural topic model permits users to incorporate document-level metadata into the topic model.

    “The structural topic model allows researchers to flexibly estimate a topic model that includes document-level metadata.”
    • Also stated in: § 1. Introduction · plain
      “the structural topic model's key innovation is that it permits users to incorporate arbitrary metadata, defined as infor-mation about each document, into the topic model.”
    • Also stated in: § 3.3. Estimate: Estimating the structural topic model · plain
      “The key innovation of the STM is that it incorporates metadata into the topic modeling framework.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    Outputs of the STM can be used to conduct hypothesis testing about relationships between topics and document metadata.

    “Outputs of the model can be used to conduct hypothesis testing about these relationships.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 65
  • May combine two claims. 21
  • Not yet classified. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Topic 18 discusses the Bush administration.

    “Topic 18 discusses the Bush administration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topics 12 (Iraq War) and 20 (Bush administration) are connected in the correlation graph.

    “We can use the correlation graph to observe the connection between Topics 12 (Iraq War) and 20 (Bush administration).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Better solutions (as measured by higher held-out log-likelihood) are found with fewer iterations and a faster runtime per iteration.

    “Better solutions (as measured by higher stm: Structural Topic Models in R held-out log-likelihood) are found with fewer iterations and a faster runtime per iteration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topic 6 is strongly used slightly more by liberals as compared to conservatives.

    “We see Topic 6 is strongly used slightly more by liberals as compared to conservatives”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Sarah Palin/vice president Topic 13 is a relatively minor proportion of the discourse.

    “We see, for example, that the Sarah Palin/vice president Topic 13 is actually a relatively minor proportion of the discourse.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topic 18, the discussion of Bush, was largely associated with liberal writers.

    “Topic 18, the discussion of Bush, was largely associated with liberal writers”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The most common topic is a general topic full of words that bloggers commonly use, and therefore is not very interpretable.

    “The most common topic is a general topic full of words that bloggers commonly use, and therefore is not very interpretable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In extensive testing, the authors have come across very few cases where the sigma.prior argument was needed.

    “However, in extensive testing we have come across very few cases where this was needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topic 13 peaks when Sarah Palin became John McCain's running mate at the end of August in 2008.

    “The topic peaks when Sarah Palin became John McCain's running mate at the end of August in 2008.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Liberals discussed Topic 20 (Bush administration) a great deal.

    “whereas liberals discussed this topic a great deal”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topic 6 describes discussions of the Obama and McCain campaigns during the 2008 presidential election.

    “We see that Topic 6 describes discussions of the Obama and McCain campaigns during the 2008 presidential election.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Conservatives never wrote much about Topic 20 (Bush administration).

    “We observe that conservatives never wrote much about this topic”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The 'Jeffreys' option can be much slower than the default option.

    “We caution that this can be much slower than the default option.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Topic 10 is related to Guantanamo.

    “Topic 10 is related to Guantanamo.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Topic 13 is close to the middle but still conservative-leaning.

    “while Topic 13 is close to the middle but still conservative-leaning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Over time, Topic 20 (Bush administration) diminished in salience.

    “but over time the topic diminished in salience.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    STM with random initialization took 7.2 minutes and 280 iterations to converge, averaging less than 1.5 seconds per iteration.

    “It took 7.2 minutes and 280 iterations to converge, averaging less than 1.5 seconds per iteration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The initialization is deterministic but may not produce exactly the same results across machines due to differences in numerical precision; it will always produce the same result within a machine for a given version of stm.

    “While the initialization is deterministic, we have observed in some circumstances it may not produce exactly the same results across machines due to differences in numerical precision.It will, however, always produce the same result within a machine for a given version of stm.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Under default convergence settings, STM with random initialization had an expected held-out log-likelihood of -6.92, better than the CTM implementation.

    “Under default convergence settings the expected held-out log-likelihood was -6.92; better than the CTM implementation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    At this point you can only have a single variable as a content covariate, although that variable can have any number of groups; it cannot be continuous.

    “At this point you can only have a single variable as a content covariate, although that variable can have any number of groups.It cannot be continuous.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The computational cost of this type of model rises quickly with the number of groups.

    “Note that the computational cost of this type of model rises quickly with the number of groups and so it may be advisable to keep it small.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package provides tools for machine-assisted reading of text corpora.

    “which provides tools for machine-assisted reading of text corpora and is available from the Comprehensive R Archive Network (CRAN) at https://CRAN.R-project.org/package=stm.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The txtorg program generates three separate files: a metadata file, a vocabulary file, and a file with the original documents.

    “The txtorg program generates three separate files: a metadata file, a vocabulary file, and a file with the original documents.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lucas et al. 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package provides tools to facilitate the work flow associated with analyzing textual data.

    “The stm package also provides tools to facilitate the work flow associated with analyzing textual data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package's textProcessor function can convert and process spreadsheet data to make it ready for analysis in the stm package.

    “If the researcher reads this data into a R data frame, then the stm package's function textProcessor can conveniently convert and processes the data to make it ready for analysis in the stm package.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package represents a text corpus in three parts: a documents list, a vocab character vector, and a metadata matrix.

    “The stm package represents a text corpus in three parts: a documents list containing word indices and their associated counts, 4 a vocab character vector containing the words associated with the word indices, and a metadata matrix containing document covariates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Function readCorpus is capable of loading data from a wide variety of formats, including the standard R matrix class and sparse matrices from slam and Matrix.

    “Function readCorpus is capable of loading data from a wide variety of formats, including the standard R matrix class along with the sparse matrices from the packages slam and Matrix (Bates and Maechler 2019).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Estimation of the structural topic model is accomplished through a fast variational approximation.

    “Estimation is accomplished through a fast variational approximation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    The STM is a generative model of word counts.

    “Like other topic models, the STM is a generative model of word counts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The design of the stm package gives users a broad array of options to process raw text data, explore and analyze the data, and present findings using a variety of plotting tools.

    “The design of the package is such that users have a broad array of options to process raw text data, explore and analyze the data, and present findings using a variety of plotting tools.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The STM can be used to model framing of international newspapers, open-ended survey responses, online class forums, Twitter feeds and religious statements, lobbying reports, and other text sources.

    “With the STM, users can model the framing of international newspapers (Roberts, Stewart, and Airoldi 2016b), open-ended survey responses in the American National Election Study (Roberts et al. 2014), online class forums (Reich, Tingley, Leder-Luis, Roberts, and Stewart 2015), Twitter feeds and religious statements (Lucas, Nielsen, Roberts, Stewart, Storer, and Tingley 2015), lobbying reports (Miln”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Roberts, Stewart, and Airoldi 2016b

      Not checked in the free run.

    • cites Roberts et al. 2014

      Not checked in the free run.

    • cites Reich, Tingley, Leder-Luis, Roberts, and Stewart 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Regularizing prior distributions are used for γ, κ, and optionally Σ, which helps to enhance interpretation and prevent overfitting.

    “Regularizing prior distributions are used for γ, κ, and (optionally) Σ, which helps to enhance interpretation and prevent overfitting.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The STM allows using topical prevalence covariates, a topical content covariate, both, or neither.

    “It is important to note, however, that the model allows using topical prevalence covariates, a topical content covariate, both, or neither.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The added randomness from the projection means that the Lee and Mimno algorithm is not deterministic like the standard 'Spectral' initialization type.

    “The added randomness from the projection means that the algorithm is not deterministic like the standard”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Mimno et al. (2011) show that semantic coherence correlates well with human judgment of topic quality.

    “Mimno et al. (2011) show that the metric correlates well with human judgment of topic quality.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mimno et al. 2011

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The authors propose to measure topic quality through a combination of semantic coherence and exclusivity of words to topics.

    “We thus propose to measure topic quality through a combination of semantic coherence and exclusivity of words to topics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The corpus package provides another fast method for creating a corpus from raw text inputs.

    “Finally the corpus package (Perry 2017) provides another fast method for creating a corpus from raw text inputs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Perry 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Our code uses an approximation which builds off of later work by Blei's group (Wang and Blei 2013).

    “Our code uses an approximation which builds off of later work by Blei's group (Wang and Blei 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wang and Blei 2013

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    In general, SAGE topics have fewer unique words that distinguish one topic from another, but those words are more likely to be meaningful.

    “In general, this means that SAGE topics have fewer unique words that distinguish one topic from another, but those words are more likely to be meaningful.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Eisenstein et al. 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The readCorpus() function can read LDA-C sparse matrix format data.

    “The readCorpus() function can read in data of this type using the”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The labelTopics function prints words associated with each topic to the console.

    “Both these functions will print words associated with each topic to the console.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    If a document is completely removed due to pre-processing, prepDocuments will drop the corresponding row in the metadata.

    “If a document is completely removed due to pre-processing (for example because it contained only rare words), then prepDocuments will drop the corresponding row in the metadata as well.”
    Quote in the paperNot a substantial claim

  • Not yet classified.

    Interpretation

    stm provides two powerful and fast initialization strategies.

    “By contrast stm provides two powerful and fast initialization strategies as described above in Section 4.1.”
    Quote in the paperNot yet classified

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The association of Topic 18 with liberal writers is in line with the observed trend of conservatives distancing from Bush after his presidency.

    “which is in line with the observed trend of conservatives distancing from Bush after his presidency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package provides a number of options to investigate the output of the model.

    “To investigate the output of the model, the stm package provides a number of options.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    While computationally more expensive, SAGE can sometimes produce higher quality topics.

    “While computationally more expensive, this can sometimes produce higher quality topics .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Eisenstein et al. 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    STM assumes a fixed user-specified number of topics.

    “STM assumes a fixed user-specified number of topics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The Lee and Mimno procedure has no particular statistical guarantees and should not be seen as estimating the 'true' number of topics.

    “We emphasize that this procedure has no particular statistical guarantees and should not be seen as estimating the”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Semantic coherence is maximized when the most probable words in a given topic frequently co-occur together.

    “It is maximized when the most probable words in a given topic frequently co-occur together.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The SAGE framework makes it straightforward to add covariate effects into the content portion of the model.

    “Importantly for our purposes, the SAGE framework makes it straightforward to add covariate effects into the content portion of the model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The findThoughts function prints documents highly associated with each topic.

    “This function will print the documents highly associated with each topic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    prepDocuments will re-index all metadata/document relationships if any changes occur due to processing.

    “Importantly, prepDocuments will also re-index all metadata/document relationships if any changes occur due to processing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    The searchK function performs several automated tests to help choose the number of topics including calculating the held-out log-likelihood and performing a residual analysis.

    “The function will perform several automated tests to help choose the number of topics including calculating the held-out log-likelihood (Wallach, Murray, Salakhutdinov, and Mimno 2009) and performing a residual analysis (Taddy 2012).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Wallach, Murray, Salakhutdinov, and Mimno 2009

      Not checked in the free run.

    • cites Taddy 2012

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These results are particularly surprising given that the variational approximation used by STM is more involved than the one used in Blei and Lafferty (2007) and implemented in topicmodels.

    “These results are particularly surprising given that the variational approximation used by STM is more involved than the one used in Blei and Lafferty (2007) and implemented in topicmodels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Blei and Lafferty (2007)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The textProcessor function implements stemming, dropping punctuation, and stop word removal across multiple languages by using the tm package.

    “The textProcessor function implements each of these steps across multiple languages by using the tm package.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Whereas LDA tends to assign many rare words to a topic, the regularization of the SAGE model ensures that words load onto topics only when they have sufficient counts to overwhelm the prior.

    “Whereas LDA tends to assign many rare words (words that appear only a few times in the corpus) to a topic, the regularization of the SAGE model ensures that words load onto topics only when they have sufficient counts to overwhelm the prior.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Eisenstein et al. 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Semantic coherence is a criterion developed by Mimno et al. and is closely related to pointwise mutual information.

    “Semantic coherence is a criterion developed by Mimno, Wallach, Talley, Leenders, and McCallum (2011) and is closely related to pointwise mutual information (Newman, Lau, Grieser, and Baldwin 2010):”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Mimno, Wallach, Talley, Leenders, and McCallum (2011)

      Not checked in the free run.

    • cites Newman, Lau, Grieser, and Baldwin (2010)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The algorithm of Lee and Mimno (2014) projects the matrix into a low dimensional space using t-distributed stochastic neighbor embedding and then exactly solves for the convex hull.

    “The algorithm of Lee and Mimno (2014) projects the matrix into a low dimensional space using t-distributed stochastic neighbor embedding (Van der Maaten 2014) and then exactly solves for the convex hull.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Lee and Mimno (2014)

      Not checked in the free run.

    • cites Van der Maaten 2014

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    In Roberts et al. (2014) the authors noted that attaining high semantic coherence was relatively easy by having a few topics dominated by very common words.

    “In Roberts et al. (2014) we noted that attaining high semantic coherence was relatively easy by having a few topics dominated by very common words.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Roberts et al. 2014

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Spectral initialization is deterministic and globally consistent under reasonable conditions.

    “The first is to use an initialization based on the method of moments, which is deterministic and globally consistent under reasonable conditions (Arora et al. 2013;Roberts et al. 2016a).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Arora et al. 2013

      Not checked in the free run.

    • cites Roberts et al. 2016a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The Lee and Mimno algorithm has the advantage of automatically selecting the number of topics.

    “This has the advantage of automatically selecting the number of topics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Each document is modeled as a mixture of multiple topics.

    “Each document is modeled as a mixture of multiple topics.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In general, the spectral initialization outperforms LDA which in turn outperforms random initialization.

    “In general, the spectral initialization outperforms LDA which in turn outperforms random initialization.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roberts et al. (2016a)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The core idea of the spectral initialization is to approximately find the vertices of the convex hull of the word co-occurrences.

    “The core idea of the spectral initialization is to approximately find the vertices of the convex hull of the word co-occurrences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The answers the estimation procedure comes up with may depend on starting values of the parameters.

    “Put differently, the answers the estimation procedure comes up with may depend on starting values of the parameters (e.g., the distribution over words for a particular topic).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: For all mixed-membership topic models, the posterior is intractable and non-convex, creating a multi-modal estimation problem that can be sensitive to initialization.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Covariates in topical content allow the observed metadata to affect the word rate use within a given topic.

    “Covariates in topical content allow the observed metadata to affect the word rate use within a given topic -that is, how a particular topic is discussed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The stm package facilitates two approaches to dealing with initialization sensitivity.

    “There are two approaches to dealing with this that the stm package facilitates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    There is not a 'right' answer to the number of topics that are appropriate for a given corpus.

    “There is not a”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Grimmer and Stewart 2013

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Metadata covariates for topical prevalence allow the observed metadata to affect the frequency with which a topic is discussed.

    “Metadata covariates for topical prevalence allow the observed metadata to affect the frequency with which a topic is discussed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    In STM, metadata can be entered in the topic model in two ways: topical prevalence and topical content.

    “In STM, metadata can be entered in the topic model in two ways: topical prevalence and topical content.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The SAGE model conceptualizes topics as sparse deviations from a corpus-wide baseline.

    “The sparse additive generative (SAGE) model conceptualizes topics as sparse deviations from a corpus-wide baseline (Eisenstein et al. 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Eisenstein et al. 2011

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    If residuals are overdispersed, it could be that more topics are needed to soak up some of the extra variance.

    “As described in Taddy (2012), if the residuals are overdispersed, it could be that more topics are needed to soak up some of the extra variance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Taddy (2012)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Topic estimation is fundamentally imprecise, as estimation in topic model space requires both an a priori number of topics input by the user, and an optimization in a space with multiple solutions.

    “Topic estimation is fundamentally imprecise, as estimation in topic model space requires both an a priori number of topics input by the user, and furthermore an optimization in a space with multiple solutions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Held-out likelihood helps the user assess the model's prediction performance.

    “Similar to cross-validation, when some of the data is removed from estimation and then later used for validation, the held-out likelihood helps the user assess the model's prediction performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The stm package contains a variety of tools that can be used to evaluate the quality of the model as well as the user's choice of number of topics and of metadata selected for inclusion.

    “The stm package contains a variety of tools that can be used to evaluate the quality of the model as well as the user's choice of number of topics and of metadata selected for inclusion.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The stm package allows researchers to understand which variables are linked with different features of text within the topic modeling framework.

    “In doing so it allows researchers understand which variables are linked with different features of text within the topic modeling framework.”
    Quote in the paperSubstantial claim
76 of 76

  • The paper relies on this without support.

    For all mixed-membership topic models, the posterior is intractable and non-convex, creating a multi-modal estimation problem that can be sensitive to initialization.

    “As with all mixed-membership topic models, the posterior is intractable and non-convex, which creates a multi-modal estimation problem that can be sensitive to initialization.”

  • The paper relies on this without support.

    Bayesian inference for the structural topic model is challenging and computationally intensive due to the non-conjugate model structure.

    “Due to the non-conjugate model structure, Bayesian inference for the structural topic model is challenging and computationally intensive.”

  • The paper relies on this without support.

    topicmodels only allows for a model to be initialized randomly or by a pre-existing model.

    “topicmodels only allows for a model to be initialized randomly or by a pre-existing model.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Iraq War and Bush topics are numbered inconsistently

    Minor

    For the same fitted model, the paper identifies the Iraq War and Bush presidency as Topics 16 and 18 in one passage, but calls them Topics 12 and 20 in another.

  • Alternative CTM benchmark uses 100 topics instead of 20

    Minor

    The paper says the performance comparison uses 20 topics, but the command for the alternative CTM model fits 100. This makes that benchmark result inconsistent with the stated comparison setup.

  • Reported average iteration time is too low

    Minor

    A runtime of 7.2 minutes over 280 iterations averages about 1.54 seconds per iteration, not less than 1.5 seconds.

  • Figure 10 is assigned different topic numbers

    Minor

    The text describes Figure 10 as showing Topic 20, while its caption says it plots Topic 16.

4 of 4

The citation lists haven’t been fetched yet

References 73 · Cited by 1,714

Research works printed but not linked: 29

  1. Ahmed A, Xing EP (2007). "Seeking the Truly Correlated Topic Posterior -On Tight Ap- proximate Inference of Logistic-Normal Admixture Model." In M Meila, X Shen (eds.), Proceedings of the Eleventh International Conference on Artificial Intelligence and Statis- tics, volume 2 of Proceedings of Machine Learning Research, pp. 19-26. PMLR, San Juan, Puerto Rico. URL http://proceedings.mlr.press/v2/ahmed07a.html.
  2. Ahmed A, Xing EP (2010). "Timeline: A Dynamic Hierarchical Dirichlet Process Model for Recovering Birth/Death and Evolution of Topics in Text Stream." In Proceedings of the 26th International Conference on Conference on Uncertainty in Artificial Intelligence, pp. 20-29.
  3. Arora S, Ge R, Halpern Y, Mimno D, Moitra A, Sontag D, Wu Y, Zhu M (2013). "A Practical Algorithm for Topic Modeling with Provable Guarantees." In S Dasgupta, D McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 280-288. PMLR, Atlanta. URL http: //proceedings.mlr.press/v28/arora13.html.
  4. Asuncion A, Welling M, Smyth P, Teh YW (2009). "On Smoothing and Inference for Topic Models." In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intel- ligence, UAI '09, pp. 27-34. AUAI Press, Arlington. URL http://dl.acm.org/citation. cfm?id=1795114.1795118.
  5. Bischof J, Airoldi E (2012). "Summarizing Topical Content with Word Frequency and Ex- clusivity." In J Langford, J Pineau (eds.), Proceedings of the 29th International Conference on Machine Learning, ICML '12, pp. 201-208. Omnipress, New York.
  6. Blei DM, Ng A, Jordan M (2003). "Latent Dirichlet Allocation." Journal of Machine Learning Research, 3, 993-1022.
  7. Csardi G, Nepusz T (2006). "The igraph Software Package for Complex Network Research." InterJournal, Complex Systems, 1695(5), 1-9. URL http://igraph.org.
  8. Eisenstein J, Ahmed A, Xing EP (2011). "Sparse Additive Generative Models of Text." In Proceedings of the 28th International Conference on Machine Learning, pp. 1041-1048.
  9. Eisenstein J, O'Connor B, Smith NA, Xing EP (2010). "A Latent Variable Model for Geo- graphic Lexical Variation." In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 1277-1287. Association for Computational Linguistics.
  10. Foulds J, Kumar S, Getoor L (2015). "Latent Topic Networks: A Versatile Probabilistic Pro- gramming Framework for Topic Models." In International Conference on Machine Learning, pp. 777-786.
  11. Gerrish S, Blei D (2012). "How They Vote: Issue-Adjusted Models of Legislative Behavior." In Advances in Neural Information Processing Systems 25, pp. 2762-2770.
  12. Hausser J, Strimmer K (2009). "Entropy Inference and the James-Stein Estimator, with Ap- plication to Nonlinear Gene Association Networks." Journal of Machine Learning Research, 10, 1469-1484.
  13. Hoffman MD, Blei DM, Wang C, Paisley JW (2013). "Stochastic Variational Inference." Journal of Machine Learning Research, 14(1), 1303-1347.
  14. Hughes MC, Sudderth E (2013). "Memoized Online Variational Inference for Dirichlet Process Mixture Models." In CJC Burges, L Bottou, M Welling, Z Ghahramani, KQ Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 1133-1141.
  15. Lee M, Mimno D (2014). "Low-Dimensional Embeddings for Interpretable Anchor-Based Topic Inference." In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1319-1328. Association for Computational Linguistics, Doha, Qatar. URL http://www.aclweb.org/anthology/D14-1138.
  16. Milner HV, Tingley DH (2015). Sailing the Water's Edge: Domestic Politics and American Foreign Policy. Princeton University Press.
  17. Mimno D, McCallum A (2008). "Topic Models Conditioned on Arbitrary Features with Dirichlet-Multinomial Regression." In Proceedings of the Twenty-Fourth Conference Annual Conference on Uncertainty in Artificial Intelligence, pp. 411-418. AUAI Press.
  18. Mimno D, Wallach HM, Talley E, Leenders M, McCallum A (2011). "Optimizing Semantic Coherence in Topic Models." In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '11, pp. 262-272. Association for Computational Lin- guistics, Stroudsburg. URL http://dl.acm.org/citation.cfm?id=2145432.2145462.
  19. Newman D, Lau JH, Grieser K, Baldwin T (2010). "Automatic Evaluation of Topic Co- herence." In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 100-108. Associ- ation for Computational Linguistics.
  20. Roberts ME, Stewart BM, Tingley D, Airoldi EM (2013). "The Structural Topic Model and Applied Social Science." In Advances in Neural Information Processing Systems Workshop on Topic Models: Computation, Application, and Evaluation. Neural Information Process- ing Society.
  21. Romney D, Stewart B, Tingley D (2015). "Plain Text: Transparency in Computer-Assisted Text Analysis." Qualitative & Multi-Method Research, 13(1), 32-37.
  22. Socher R, Gershman S, Perotte A, Sederberg P, Blei D, Norman K (2009). "A Bayesian Analysis of Dynamics in Free Recall." Advances in Neural Information Processing Systems, 22, 1714-1722.
  23. Taddy MA (2012). "On Estimation and Selection for Topic Models." In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics.
  24. Van der Maaten L (2014). "Accelerating t-SNE Using Tree-Based Algorithms." The Journal of Machine Learning Research, 15(1), 3221-3245.
  25. Wallach HM, Murray I, Salakhutdinov R, Mimno D (2009). "Evaluation Methods for Topic Models." In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1105-1112.
  26. Wang C, Blei DM (2013). "Variational Inference in Nonconjugate Models." The Journal of Machine Learning Research, 14(1), 1005-1031. doi:10.1049/pbhe002e_ch9. · DOI
  27. Wickham H (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag. doi: 10.1007/978-3-319-24277-4. · DOI
  28. Zhao T, Liu H, Roeder K, Lafferty J, Wasserman L (2012). "The huge Package for High- Dimensional Undirected Graph Estimation in R." The Journal of Machine Learning Re- search, 13(1), 1059-1062.
  29. Dustin Tingley Department of Government Harvard University 1737 Cambridge St, Cambridge, MA, United States of America E-mail: dtingley@gov.harvard.edu URL: http://scholar.harvard.edu/dtingley Journal of Statistical Software http://www.jstatsoft.org/ published by the Foundation for Open Access Statistics http://www.foastat.org/ October 2019, Volume 91, Issue 2 Submitted: 2016-01-19 doi:10.18637/jss.v091.i02 Accepted: 2018-09-03

Authors
Roberts, M. E., Stewart, B. M., & Tingley, D.
Affiliations
  • University of California San Diego (US)
  • Princeton University (US)
  • Harvard University (US)
ORCID
Venue
Journal of Statistical Software
Publisher
Foundation for Open Access Statistic
Published
2019
Volume
91
Issue
2
ISSN
1548-7660 (ISSN-L)
DOI
10.18637/jss.v091.i02
OpenAlex
W2223092947
Semantic Scholar
bc0123c42ac968a54c511207492662c01f4a21c1
Cited by
1,714
References
73
Citations by year
2026: 252 · 2025: 359 · 2024: 258 · 2023: 268 · 2022: 183 · 2021: 184 · 2020: 116 · 2019: 36 · 2018: 24 · 2017: 20 · 2016: 8 · 2015: 4 · 2014: 2
FWCI
382.04
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Statistical Software (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Science Foundation
  • Spencer Foundation
  • William and Flora Hewlett Foundation
  • National Institutes of Health
  • Eunice Kennedy Shriver National Institute of Child Health and Human Development (P2CHD047879)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Methods and Mixture Models
Keywords
structural topic modeling, topic modeling, R package, variational approximation, uncertainty estimation, topic visualization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review