Discovery of Treatments from Text Corpora
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
An extensive literature in computational social science examines how features of messages, advertisements, and other corpora affect individuals' decisions, but these analyses must specify the relevant features of the text before the experiment.Automated text analysis methods are able to discover features of text, but these methods cannot be used to obtain the estimates of causal effects-the quantity of interest for applied researchers.We introduce a new experimental design and statistical model to simultaneously discover treatments in a corpora and estimate causal effects for these discovered treatments.We prove the conditions to identify the treatment effects of texts and introduce the supervised Indian Buffet process to discover those treatments.Our method enables us to discover treatments in a training set using a collection of texts and individuals' responses to those texts, and then estimate the effects of these interventions in a test set of new texts and survey respondents.We apply the model to an experiment about candidate biographies, recovering intuitive features of voters' decisions and revealing a penalty for lawyers and a bonus for military service.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Statistical Analysis of Compositional Data
Aitchison, J. (1986) · doi:10.1007/978-94-009-4109-0 · Cited by 4,383
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 1×
“We selected ten treatments using a combination of guidance from the sIBP, assessment using CE scores, and our own qualitative assessment of the models (Grimmer and Stewart, 2013).”
MethodSelf-citationHainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014) · Political Analysis · doi:10.1093/pan/mpt024 · Cited by 1,990 · cited 2×
“In this paper we focus on estimating the Average Marginal Component Specific Effect for factor k (AMCE k ) (Hainmueller et al., 2014).”
BackgroundMethodPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · cited 2×
“The use of the training and test set ensures that this discovery does not come at the expense of credibly inferring causal effects, insulating the research design from concerns about “p-hacking” and overfitting (Ioanni-dis, 2005; Humphreys et al., 2013; Franco et al., 2014).”
BackgroundPopulation-Based Survey Experiments
Mutz, D. C. (2011) · doi:10.1515/9781400840489 · Cited by 674
Cited by
Machine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Self-citationCausal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
“Such concerns could be avoided by randomly assigning texts to readers (Fong and Grimmer, 2016, 2021), but that may be impractical.”
BackgroundSelf-citationThe Anchoring Effect in Decision-Making with Visual Analytics
Cho, I., Wesslen, R., Karduni, A., Santhanam, S., Shaikh, S., & Dou, W. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585665 · Cited by 88
“Fong and Grimmer [18] note that topic models are susceptible to problems in estimating marginal effects due to the zero-sum properties of topic proportions.”
BackgroundIdentification, Interpretability, and
Lauretig, A. (2019) · Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science · doi:10.18653/v1/w19-2102 · Cited by 9
“Finally, the third track for future word will involve tying the anchoring approach discussed above with the emerging literature on making casual claims from text (Fong and Grimmer, 2016; Mozer et al., 2018), and taking advantage of the word similarities to identify appropriate linguistic counterfactuals.”
Background
Metadata
- Authors
- Fong, C., & Grimmer, J.
- Venue
- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2016
- Pages
- 1600-1609
- OpenAlex
- W2513458147
- Cited by
- 49
- References
- 5
- FWCI
- 17.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Discovery of Treatments from Text Corpora. https://app.cochat.ai/papers/10.18653/v1/p16-1151
- The paper
- Fong, C., & Grimmer, J. (2016). Discovery of Treatments from Text Corpora. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/p16-1151
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.