Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Quality
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Matching for causal inference is a well-studied problem, but standard methods fail when the units to match are text documents: the high-dimensional and rich nature of the data renders exact matching infeasible, causes propensity scores to produce incomparable matches, and makes assessing match quality difficult. In this paper, we characterize a framework for matching text documents that decomposes existing methods into (1) the choice of text representation and (2) the choice of distance metric. We investigate how different choices within this framework affect both the quantity and quality of matches identified through a systematic multifactor evaluation experiment using human subjects. Altogether, we evaluate over 100 unique text-matching methods along with 5 comparison methods taken from the literature. Our experimental results identify methods that generate matches with higher subjective match quality than current state-of-the-art techniques. We enhance the precision of these results by developing a predictive model to estimate the match quality of pairs of text documents as a function of our various distance scores. This model, which we find successfully mimics human judgment, also allows for approximate and unsupervised evaluation of new procedures in our context. We then employ the identified best method to illustrate the utility of text matching in two applications. First, we engage with a substantive debate in the study of media bias by using text matching to control for topic selection when comparing news articles from thirteen news sources. We then show how conditioning on text data leads to more precise causal inferences in an observational study examining the effects of a medical intervention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Austin, P. C. (2009) · Statistics in Medicine · doi:10.1002/sim.3697 · Cited by 6,758 · cited 2×
“Unfortunately, determinations of what constitutes “suitable” balance or match quality are often based on arbitrary criteria (Imai et al., 2008; Austin, 2009), and assessing whether a matching procedure has been successful can be quite difficult.”
BackgroundBlei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · cited 2×
“A.1.2 Representations based on statistical topic models An alternative representation for text, popular in the text analysis literature, is based on statistical topic models (Blei, 2012), e.g., LDA (Blei et al., 2003) and STM (Roberts et al., 2016).”
BackgroundMethodInfluentialMatching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · cited 6×
“ilitate causal inferences about the eects of a particular treatment, action, or intervention from non-randomized data in the presence of confounding covariates (Rubin,1973b;Rosenbaum,2002;Rubin,2006;Stuart, 2010). The principles behind matching can also be used to create sharp, targeted comparisons of units in order t…”
BackgroundMethodInfluentialStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“To make principled comparisons between groups of documents within the corpus, we borrow from the notation and principles of the Rubin Causal Model (RCM) (Holland, 1986).”
MethodCausal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · cited 1×
“Online crowd-sourcing platforms such as these have been shown to be effective for similarity evaluations in a number of settings (Mason and Suri, 2012).”
BackgroundCheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Design of Observational Studies
Rosenbaum, P. R. (2010) · Springer Series in Statistics · doi:10.1007/978-1-4419-1213-8 · Cited by 1,150
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047
Matching to remove bias in observational studies
Rubin, D. B. (1973) · Biometrics · doi:10.2307/2529684 · Cited by 737
Cited by
Machine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
“Thus far, applications have largely focused on other domains, such as biomedical data, where trustworthiness is particularly critical (e.g., Muandet et al., 2013), and computer vision, where there is a strong interest in cross-domain learning (e.g., Ghifary et al., 2015) and where it is relatively…”
BackgroundMethodInfluentialIdentification, Interpretability, and
Lauretig, A. (2019) · Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science · doi:10.18653/v1/w19-2102 · Cited by 9
“Finally, the third track for future word will involve tying the anchoring approach discussed above with the emerging literature on making casual claims from text (Fong and Grimmer, 2016; Mozer et al., 2018), and taking advantage of the word similarities to identify appropriate linguistic counterfactuals.”
BackgroundCombs, A., Tierney, G., Guay, B., Merhout, F., Bail, C. A., Hillygus, D. S., & Volfovsky, A. (2022) · doi:10.31235/osf.io/cwgu5 · Cited by 5
Metadata
- Authors
- Mozer, R., Miratrix, L., Kaufman, A. R., & Jason Anastasopoulos, L.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2020-03-17
- Volume
- 28
- Issue
- 4
- Pages
- 445-468
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W2781567951
- Cited by
- 67
- References
- 14
- FWCI
- 5.53
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.