CoChat
Sign Up
Cited by 71Computational and Text Analysis MethodsTopic ModelingSentiment Analysis and Opinion Mining

Embedding Regression: Models for Context-Specific Description and Inference

Open access
RODRIGUEZ, PEDRO L., SPIRLING, ARTHUR, STEWART, BRANDON M.·2023·American Political Science Review
DOI 10.1017/s0003055422001228
Open this paper in CoChat — free accountOpen PDF

Abstract

Social scientists commonly seek to make statements about how word use varies over circumstances—including time, partisan identity, or some other document-level covariate. For example, researchers might wish to know how Republicans and Democrats diverge in their understanding of the term “immigration.” Building on the success of pretrained language models, we introduce the à la carte on text (conText) embedding regression model for this purpose. This fast and simple method produces valid vector representations of how words are used—and thus what words “mean”—in different contexts. We show that it outperforms slower, more complicated alternatives and works well even with very few documents. The model also allows for hypothesis testing and statements about statistical significance. We demonstrate that it can be used for a broad range of important tasks, including understanding US polarization, historical legislative development, and sentiment detection. We provide open-source software for fitting the model.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces conText, an à la carte on text (ALC) embedding regression model for context-specific word meaning and hypothesis testing. It shows the method is fast, sample-efficient, and competitive with slower alternatives across US partisan speech, UK-US “empire” usage, and Brexit sentiment.

Key findings

  • In a replication of Rodman (2020), ALC tracks the chronological model and produces more semantically interpretable nearest neighbors; the first 25-year slice has only 71 documents, and almost 30% of word-slice combinations have fewer than 10 observations. The ALC process takes only a few milliseconds, versus about five hours of compute time on an eight-core machine for the replication.
  • ALC can distinguish word senses from single instances: using 400 realizations of trump/Trump from a New York Times corpus, k-means cluster homogeneity was close to RoBERTa; simple averaging also performed well, but ALC’s nearest neighbors aligned with meanings (Trump with president Trump; trump with card-game/defeating senses), whereas simple averaging produced mainly stopwords.
  • In Congressional Record Sessions 111–114, party differences were larger than gender differences, and immigration was the “most partisan” target; all coefficients were statistically significant at the 0.01 level. Nearest neighbors showed Democrats closer to reform/bipartisan and Republicans closer to enforce/illegals.
  • For “empire,” US and UK understandings diverged around 1949–50, confirmed by a structural break test; post-break US nearest neighbors included invasion, Soviet, communists, and Lithuania, while UK usage continued with British empire, India, and Rhodesia. In the Brexit sentiment case, a dictionary approach had only 56% of terms occur and 69% of speeches score 0, while 99% of dictionary terms appeared in pretrained embeddings, allowing all speeches to be scored.

Methods

  • The method builds on ALC embeddings: pretrained embeddings plus the context window around a focal word are averaged and transformed by a learned matrix A. The conText regression treats each instance embedding as an outcome \(Y\) (\(n \times D\)), with covariates \(X\) and coefficients \(\beta\); permutation shuffling gives empirical p-values, and nonparametric bootstrap gives standard errors/confidence intervals.
  • Data and use cases include a New York Times lead-paragraph corpus for Trump/trump and pre/post-2016 election tests; Congressional Record Sessions 111–114 for abortion, immigration, marriage, and stopwords; Hansard versus Congressional Record 1935–2010 for “empire”; and UK House of Commons debate text 2001–2019 for Brexit sentiment on education, nhs, and eu.
  • Analyses include k-means clustering with PCA for sense separation, multivariate multiple regression with party/gender/NOMINATE covariates, cosine similarity nearest neighbors, a structural break test (Bai and Perron 1998), and sentiment measured via inner products with the Warriner, Kuperman, and Brysbaert (2013) dictionary.

Limitations

  • Stated: ALC assumes the meaning of nonfocal words is essentially constant; the method estimates high-dimensional means from few data points; estimates of norms have finite-sample bias for rare words, so care is needed when comparing words or groups with substantially different amounts of data.
  • Stated: users still choose context window size and preprocessing; experiments suggest avoiding windows smaller than 5, that windows of 6 and 12 perform similarly, and that stopword removal makes little difference. Uncertainty in the A matrix is described as minimal, and changing contexts over time is a second-order concern for texts from the past 100 years or so.
  • Evident from the text: the approach is descriptive and structuralist, making no causal claims that distributions cause human understandings; substantive interpretation and validation are left to the researcher.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1017/s0003055422001228

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

19 of 32 cited sources checked · 11 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Topic models are rarely used as a mechanism for studying how individual words are used to convey different ideas.

    “Although such models have a built-in notion of polysemy (a single word can be allocated to different topics), they are rarely used as a mechanism for studying how individual words are used to convey different ideas (Grimmer and Stewart 2013).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Grimmer and Stewart 2013 source

      Section 6.1.2 explains that LDA assigns individual words to topics and describes applications that measure attention to topics, supporting the sentence’s description of how topic models work. It does not say that topic models are rarely used to study how individual words convey different ideas; that frequency claim remains unsupported. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
  • Premisecitation

    Existing efforts to address inference in embeddings are typically data intensive and computationally intensive.

    “Although there have been efforts to address inference in embeddings (see, e.g, Kulkarni et al. 2015;Lauretig 2019), they are typically data intensive and computationally intensive.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Kulkarni et al. 2015 source
    • cites Lauretig 2019 source

      The passages available from our copy identify Lauretig’s paper, but none states that efforts to address inference in embeddings are data intensive or computationally intensive. Searches for those terms and for computational cost returned no relevant passage from the paper’s body, so the resource claim cannot be checked against the available text. (LLM, conf=0.69)

      Experimental — accuracy not yet measured
  • Premisecitation

    A large corpus can be used to estimate the transformation matrix A.

    “The idea is that a large corpus (generally the corpus the embeddings were originally trained on, such as Wikipedia) can be used to estimate the transformation matrix A.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Khodak et al. 2018 source
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).
  • Premisecitation

    If the random walk model holds, a researcher can obtain an embedding for a word by averaging the embeddings of the words around it and multiplying them by a particular square matrix A.

    “The main result from Arora et al. (2018) shows the following: if the random walk model holds, the researcher can obtain an embedding for word w (e.g., "bill") by taking the average of the embeddings of the words around w (u w Þ and multiplying them by a particular square matrix A .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Arora et al. 2018 source
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).
  • Premisecitation

    There has been relatively little work on whether the same words appear in different contexts.

    “Although there have been efforts to understand differential word rate of use within topics pertaining to these terms (e.g., Monroe, Colaresi, and Quinn 2008), there has been relatively little work on whether the same words appear in different contexts.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Monroe, Colaresi, and Quinn 2008 source
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 31
  • The quote only partly states this claim. 2

  • Classified as probably not a substantial claim.

    Findingnone given

    The approach yields novel insights about the relationship between the UK Prime Minister and his backbenchers on the European Union.

    “We build on earlier work (e.g., Osnabrügge, Hobolt, and Rodon 2021;Slapin et al. 2018) for the UK House of Commons, yielding novel insights about the relationship between the UK Prime Minister and his backbenchers on the European Union.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Osnabrügge, Hobolt, and Rodon 2021 source
    • cites Slapin et al. 2018 source

  • May combine two claims.

    Findingown data

    Latent semantic analysis does not fare well in distinguishing Trump senses, whereas ALC performs close to on par with transformer-based RoBERTa embeddings.

    “Latent sematic analysis does not fare well in this task; ALC, on the other hand, performs close to on par with transformer-based RoBERTa embeddings.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Liu et al. 2019 source

      Section 2.2 (p. 2) describes the transformer architecture used by the paper, and Section 5 (p. 6) identifies the resulting model as RoBERTa. The paper does not evaluate latent semantic analysis or ALC on distinguishing Trump senses, so it does not support the claimed performance comparison. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Documents can be conceived of as a random walk in a discourse space, where words are more likely to follow other words if they are closer in an embedding space.

    “Those papers conceive of documents being a "random walk" in a discourse space, where words are more likely to follow other words if they are closer to them in an embedding space.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Arora et al. 2016 source

      Section 2, “Generative model and its properties” (page not supplied), models text generation as a random walk of a latent discourse vector that makes nearby words arise under similar discourses. But Section 1.1, “Related work” (page not supplied), distinguishes this model from one in which the next word’s probability depends directly on its embedding-space distance from the previous word. The claim conflates those two models. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Arora et al. 2018 source

      Section 2, “Justification for Linearity Assertion” (page not provided), describes a slow random walk through discourse space that generates nearby words. But it says word probabilities depend on closeness to the current discourse vector, not directly on closeness to the preceding word’s embedding. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • May combine two claims.

    Premisecitation

    Rodman (2020) tested four different methods by comparing results to the gold standard, and the chronologically trained model (Kim et al. 2014) was the best performer.

    “Rodman (2020) tests four different methods by comparing results to the gold standard; ultimately, the chronologically trained model (Kim et al. 2014) is the best performer.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Rodman (2020) source
    • cites Kim et al. 2014 source

  • May combine two claims.

    Premisecitation

    Embedding representations can be used to solve analogy reasoning tasks, which implies that the vectors capture relational meaning between words.

    “For example, embedding representations can be used to solve analogy reasoning tasks, implying the vectors do indeed capture relational meaning between words (e.g., Arora et al. 2018;Mikolov et al. 2013).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Arora et al. 2018 source

      The Introduction and Section 4.2 discuss how embeddings represent word senses and test that structure, but they do not address solving word-analogy tasks. Searches for analogy reasoning and relational meaning found no passage in our copy supporting the cited example or its inference. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Mikolov et al. 2013

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • The quote only partly states this claim.

    Findingown data

    The approach's performance compares favorably to the industry standard (Rodman 2020) in a study of a New York Times API corpus.

    “First, we demonstrate the strength of our approach by comparing its performance to the "industry standard" as laid out by Rodman (2020) in a study of a New York Times API corpus, where slow changes over long periods are the norm.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rodman (2020) source

  • May combine two claims.

    Findingown data

    The model can be used for a broad range of important tasks, including understanding US polarization, historical legislative development, and sentiment detection.

    “We demonstrate that it can be used for a broad range of important tasks, including understanding US polarization, historical legislative development, and sentiment detection.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Supplementary Materials D, inferences from the conText regression model are similar to a full embeddings model, and the conText approach is more stable by holding constant embeddings for all nonfocal words.

    “As we demonstrate in Supplementary Materials D, inferences are similar and our approach is more stable by virtue of holding constant embeddings for all nonfocal words.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The government-versus-opposition pattern is true for education, and for health to a lesser extent.

    “This is true for education, and health to a lesser extent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Even for a relatively small corpus, Rodman's chronological model fitting process is computationally expensive, and the replication took about five hours of compute time on an eight-core machine.

    “Even for a relatively small corpus, this process is computationally expensive, and our replication took about five hours of compute time on an eight-core machine.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    In the Rodman (2020) corpus, the first 25-year slice of data has only 71 documents, and in almost 30% of the word-slice combinations there are fewer than 10 observations.

    “This is challenging because the corpus is small-the first 25-year slice of data has only 71 documents-and in almost 30% of the word-slice combinations there are fewer than 10 observations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ALC outputs nearest neighbors aligned with the meaning of each term: Trump is associated with president Trump, whereas trump is largely associated with its two related other meanings, a suit in trick-taking games and defeating someone.

    “The ALC method, on the other hand, outputs nearest neighbors aligned with the meaning of each term, Trump is associated with president Trump, whereas trump is largely associated with its two related other meanings: a suit in trick-taking games and defeating someone.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The Conservatives are the converse: both elites and backbenchers have negative valence for government policy when in opposition but are much more enthusiastic when in government.

    “The Conservatives are the converse: both elites and backbenchers have negative valence for government policy when in opposition but are much more enthusiastic when in government.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The predicted ALC embedding for immigration is closer to enforce and illegals at higher values of the NOMINATE score, closer to reform and bipartisan at lower values, and the feature amend shows similar values across the full range.

    “Consistent with our results above, we observe how the predicted ALC embedding for immigration is closer to enforce and illegals at higher values of the NOMINATE score.It is closer to reform and bipartisan at lower values.The feature amend on the other hand, shows similar values across the full range.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    ALC can learn meaningful embeddings from one example use.

    “À la carte embeddings will assist us, but first we show that it can learn meaningful embeddings from one example use.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ALC has no problem with the harder case of the meaning shift of Trump the surname after the 2016 election, returning excellent cluster homogeneity and nearest neighbors.

    “But as we show in Supplementary Materials B, ALC has no problem with this case either, returning excellent cluster homogeneity and nearest neighbors.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Trump's election and presidency constituted a major break with respect to his real-estate/celebrity past.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The ALC process for inducing period-specific embeddings is transparent and takes only a few milliseconds, with the embeddings themselves involving six matrix multiplies.

    “The entire process is transparent and takes only a few milliseconds (the embeddings themselves involve six matrix multiplies).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Simple averaging of embeddings performs surprisingly well in the Trump sense distinction task, coming out on top in the comparison.

    “Simple averaging of embeddings also performs surprisingly well, coming out on top in this comparison.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The correlation between Tory leadership and backbench sentiment is generally positive or close to zero for education and health but negative for the EU, moving in opposite directions.

    “The middle row makes this more explicit: the correlation between Tory leadership and backbench sentiment is generally positive or close to zero for education and health but negative for the EU-that is, moving in opposite directions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The inner product often delivers very similar results to conventional cosine similarity.

    “The conventional cosine similarity provides interpretable neighbors, but the inner product often delivers very similar results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Potential changes to contexts of targets is a second-order concern, at least for texts from the past 100 years or so.

    “Potential changes to contexts of targets is a second-order concern, at least for texts from the past 100 years or so.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Removing stopwords and using bigger context windows results in marginally better performance in separating the uses of trump versus Trump.

    “We found that removing stopwords and using bigger context windows results in marginally better performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For LSA, using two dimensions and tf-idf weighting produced the best results.

    “For LSA we use two dimensions and tf-idf weighting.We found these settings produced the best results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Conservative backbenchers were unsatisfied with their own government's EU policy prior to the referendum.

    “FIGURE 9. Conservative Backbenchers Were Unsatisfied with Their Own Government's EU Policy Prior to the Referendum”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The conventional cosine similarity provides interpretable neighbors.

    “The conventional cosine similarity provides interpretable neighbors, but the inner product often delivers very similar results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Stemming/lemmatization does not change results much in practice.

    “Stemming/lemmatization does not change results much in practice.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    99% of the terms in the dictionary appear in the pretrained embeddings, allowing all speeches to be scored.

    “This contrasts with the 99% of terms in the dictionary appearing in the pretrained embeddings, allowing for all speeches to be scored.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Windows of size 6 and 12 perform very similarly to each other and acceptably well in an absolute sense.

    “Windows of size 6 and 12 perform very similarly to each other and acceptably well in an absolute sense.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Removing stopwords from contexts used in estimating ALC embeddings makes very little difference to any type of performance.

    “Removing stopwords from contexts used in estimating ALC embeddings makes very little difference to any type of performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning

    Polysemy is partly why context is vital.

    “This is partly due to polysemy-the word "society" might mean many different things.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The model allows for hypothesis testing and statements about statistical significance.

    “The model also allows for hypothesis testing and statements about statistical significance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Word embeddings are a promising avenue for investigating context.

    “A promising avenue for such investigations has been the use of "word embeddings"-a family of techniques that conceive of meaning as emerging from the distribution of words that surround a term in text (e.g., Mikolov et al. 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mikolov et al. 2013

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning

    Context is vital: we cannot understand what is meant by a speaker or author without it.

    “In a basic sense, context is vital: we literally cannot understand what is meant by a speaker or author without it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisereasoning

    The issue of context is broader than polysemy and is at the core of human communication.

    “But the issue is broader than this and is at the core of human communication.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Neither of these problems are well addressed by current techniques.

    “Neither of these problems are well addressed by current techniques.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The study of context has been a long-standing endeavor in social science.

    “Unsurprisingly then, the study of context has been a long-standing endeavor in social science.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Skinner 1969 source
    • cites Austin 1962

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Geertz 1973

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisereasoning

    The measurement and modeling of context is a general problem.

    “Beyond this specific example, the measurement and modeling of context is obviously a general problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    Social scientists want to make statements about the statistical properties and relationships between embeddings.

    “Although representations themselves are helpful, social scientists want to make statements about the statistical properties and relationships between embeddings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The method is possible without substantial computation and requires only the text immediately around the word of interest.

    “This is possible without substantial computation and, in contrast to other approaches, requires only the text immediately around the word of interest.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Because the matrix A is constant, it can be swapped into the expectation and the resulting expectation conditional on a covariate X can be calculated implicitly through a multivariate regression procedure.

    “Here we note that because the matrix A is constant we can easily swap it into the expectation and then calculate the resulting expectation conditional on some covariate X : E Au w jX ½ .In particular, this can be done implicitly through a multivariate regression procedure.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The transformation matrix A is constant across contexts and instances.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The natural log is a simple choice for α(·) and works well.

    “The natural log is a simple choice for α Á ð Þ, and works well.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • May combine two claims.

    Interpretationnone given

    Unlike other methods, ALC does not need an entirely new corpus to learn embeddings for select focal words; it can select particular words and calculate only their embeddings using only the contexts around those particular words.

    “Importantly, unlike other methods, we don't need an entirely new corpus to learn embeddings for select focal words; we can select particular words and calculate (only) their embeddings using only the contexts around those particular words.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Although the magnitude of the difference between groups is not directly interpretable, one can comment on whether it is statistically significantly different from zero.

    “Although the magnitude of this difference is not directly interpretable, we can nonetheless comment on whether it is statistically significantly different from zero.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Training separate sets of embeddings for Republicans and Democrats and then realigning them is an extra computational step and may not be feasible in other use cases where the vocabularies do not have much overlap.

    “We could instead train a separate set of embeddings-one for Republicans and one for Democrats-and then realign them.This is an extra computational step and may not be feasible in other use cases where the vocabularies do not have much overlap.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    In the case of word meanings in discrete subgroups, the conText regression procedure is exactly the same as the use of ALC applied above.

    “In the case of word meanings in discrete subgroups, this is exactly the same as the use of ALC applied above.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    If desired, one can obtain the estimated sampling distribution and standard errors of the normed coefficients via nonparametric bootstrap.

    “Note that, if desired, one can obtain the estimated sampling distribution (and thus standard errors) of the (normed) coefficients via nonparametric bootstrap.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The paper uses a variant of covariate assignment shuffling suggested by Gentzkow, Shapiro, and Taddy (2019) to compute empirical p-values.

    “To do this, we use a variant of covariate assignment shuffling suggested by Gentzkow, Shapiro, and Taddy (2019).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gentzkow, Shapiro, and Taddy (2019) source

  • May combine two claims.

    Interpretationreasoning

    ALC is efficient in the use of information because once the transformation matrix is estimated, it is only necessary that u_w converges, meaning one only needs to estimate a D-dimensional mean from a set of samples; with a six-word symmetric context window there are 12 words total, giving a sample of size 12 per instance of the focal word.

    “Finally, ALC is efficient in the use of information.Once the transformation matrix is estimated, it is only necessary that u w converges-in other words, we only need to estimate a D-dimensional mean from a set of samples.In the case of a six-word symmetric context window, there are 12 words total within the context window; thus, for each instance of the focal word we have a sample of size 12 from”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Reweighting by word frequency addresses the fact that words that appear more frequently have embeddings measured with greater certainty.

    “This reweighting addresses the fact that words that appear more frequently have embeddings that are measured with greater certainty.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The United Kingdom's relative decline as a Great Power during the postwar period has been well documented.

    “The United Kingdom's relative decline as a Great Power during the postwar period has been well documented (e.g., Hennessy 1992).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hennessy 1992

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    One reading of the nearest neighbors is that Democrats were pushing for reform of existing laws, whereas Republicans were mainly arguing for enforcement.

    “One reading of these nearest neighbors is that Democrats were pushing for reform of existing laws, whereas Republicans were mainly arguing for enforcement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Top nearest contexts corroborate the reading of the nearest neighbors, suggesting some validity of the general approach.

    “We can corroborate this via the top nearest contexts-that is, the individual contexts of immigration-embedded using ALC-that are closest to each party's ALC embedding of the term (see Table 5).This suggests some validity of our general approach.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The transformation matrix is not particularly hard to learn (it is a linear regression problem), and each subsequent induced word embedding is a single matrix multiply.

    “The transformation matrix is not particularly hard to learn (it is a linear regression problem), and each subsequent induced word embedding is a single matrix multiply.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The general approach is not inextricably connected to the ALC embedding strategy.

    “But our general approach is not inextricably connected to this particular method for estimating contextually specific meanings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Rodriguez and Spirling (2022) provide extensive results on the comparison between pretrained and locally fit embeddings for current models.

    “We note that Rodriguez and Spirling (2022) provide extensive results on this comparison for current models; thus, here we are mostly extending those enquiries to our specific approach.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rodriguez and Spirling (2022) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The approach requires no active tuning of parameters, but there are still choices to make.

    “Our approach requires no active tuning of parameters, but that does not mean that there are no choices to make.”
    Quote in the paperNot a substantial claim
56 of 56

  • The paper relies on this without support.

    Topic approaches that allow systematic variation in the use of a word across topics by different pieces of observed metadata are computationally intensive.

    “And though topic approaches do exist that allow for systematic variation in the use of a word across topics by different pieces of observed metadata (Roberts, Stewart, and Airoldi 2016), they are computationally intensive (especially relative to the approaches we present below).”

  • The paper relies on this without support.

    Existing efforts to compare embeddings across groups and to give frameworks for conditional relationships are nontrivial to implement.

    “Although there have been efforts to compare embeddings across groups (Rudolph et al. 2017) and to give frameworks for such conditional relationships (Han et al. 2018), these are nontrivial to implement.”

  • The paper relies on this without support.

    A common problem in Americanist political science is to estimate partisan differences in the usage of a given term.

    “A common problem in Americanist political science is to estimate partisan differences in the usage of a given term.”

  • The paper relies on this without support.

    Social scientists want to make systematic inferences about embeddings, which requires statements about uncertainty.

    “For a variety of use cases, social scientists want to make systematic inferences about embeddings-which requires statements about uncertainty.”

  • The paper relies on this without support.

    Words that co-occur in similar contexts are similar in meaning (the distributional hypothesis).

    “The distributional hypothesis then allows us to infer that those two words are similar in meaning.”

  • The paper relies on this without support.

    Word meaning can be inferred from the contexts in which a word appears (the distributional hypothesis).

    “in the sense of the distributional hypothesis, the meaning-has shifted”

  • The paper relies on this without support.

    The transformation matrix A is constant across contexts and instances.

    “Here we note that because the matrix A is constant we can easily swap it into the expectation”

  • The paper relies on this without support.

    The meaning of Trump, the surname, underwent a significant transformation once Donald J. Trump was elected president of the United States in November 2016.

    “we also know that the meaning of Trump, the surname, underwent a significant transformation once Donald J. Trump was elected president of the United States in November 2016.”

  • The paper relies on this without support.

    Trump's election and presidency constituted a major break with respect to his real-estate/celebrity past.

    “Given Trump won the election and subsequently became President-a major break with respect to his real-estate/celebrity past-we expect a statistically significant change for Trump relative to any changes in the usage of Clinton.”

  • The authors state this as an assumption.

    ALC assumes that the meaning of nonfocal words is essentially constant.

    “First, ALC assumes that the meaning of nonfocal words is essentially constant.”

  • The authors state this as an assumption.

    The A matrix is treated as fixed.

    “Note here that we are treating the A matrix as fixed, and thus we are not incorporating uncertainty in those estimates.”
11 of 11

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Sum of coefficients mislabeled as a row of β

    Minor

    The text identifies β0 as the first row of β and elsewhere identifies β0 and β1 as its coefficients. The second row is therefore β1, not β0 + β1; the latter is the predicted embedding for the group.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Hinton, 1986, Proceedings of The Eighth Annual Conference of the Cognitive Science Society, 1
  2. Hennessy, 1992, Never Again: Britain 1945-1951
  3. Wu, 2019, (Mimeo)
  4. Geertz, 1973, Thick Description: Toward an Interpretive Theory of Culture, Turning Points in Qualitative Research: Tying Knots in a Handkerchief, 3, 143
  5. Gentzkow, 2018, Congressional Record for the 43rd–114th Congresses: Parsed Speeches and Phrase Counts

Cited by

Sorted by citations

Related

Authors
RODRIGUEZ, P. L., SPIRLING, A., & STEWART, B. M.
Affiliations
  • New York University (US)
  • Princeton University (US)
Corresponding author
  • SPIRLING, ARTHUR
  • STEWART, BRANDON M.
ORCID
Venue
American Political Science Review
Publisher
Cambridge University Press (CUP)
Published
2023-01-19
Volume
117
Issue
4
Pages
1255-1274
ISSN
  • 0003-0554 (ISSN-L)
  • 1537-5943
DOI
10.1017/s0003055422001228
OpenAlex
W4317434765
Semantic Scholar
761dc674ee0aa898043ead184a922f21731ff37a
Cited by
70
References
52
Citations by year
2026: 17 · 2025: 22 · 2024: 17 · 2023: 12 · 2022: 1 · 2021: 1
FWCI
32.88
Type
journal article
Language
en
Open access
Licence
Locations
  • American Political Science Review (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
  • Vanderbilt University
  • Princeton University
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
context, pre-trained language models, word meaning in context, political polarization, legislative developments, sentiment analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-13.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-13.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review