The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It
Open accessAbstract
Despite the popularity of unsupervised techniques for political science text-as-data research, the importance and implications of preprocessing decisions in this domain have received scant systematic attention. Yet, as we show, such decisions have profound effects on the results of real models for real data. We argue that substantive theory is typically too vague to be of use for feature selection, and that the supervised literature is not necessarily a helpful source of advice. To aid researchers working in unsupervised settings, we introduce a statistical procedure and software that examines the sensitivity of findings under alternate preprocessing regimes. This approach complements a researcher’s substantive understanding of a problem by providing a characterization of the variability changes in preprocessing choices may induce when analyzing a particular dataset. In making scholars aware of the degree to which their results are likely to be sensitive to their preprocessing decisions, it aids replication efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Foundations of statistical natural language processing
Manning, C. D., & Schütze, H. (1999) · Cited by 9,816
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · cited 1×
“Stemming refers to the process of reducing a word to its most basic form (Porter, 1980).”
BackgroundMachine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,926
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 3×
“A commonly used rule of thumb is to discard terms that appear in less than 0.5-1% of documents (Grimmer, 2010; Yano, Smith and Wilkerson, 2012; Grimmer and Stewart, 2013), however, there has been no systematic study of the effects this preprocessing choice has on downstream analyses.”
BackgroundMethodInfluentialAdvances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,701
Thumbs up? Sentiment Classification using Machine Learning Techniques
Pang, B., Lee, L., & Vaithyanathan, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205070 · Cited by 2,191
Structural Topic Models for Open‐Ended Survey Responses
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014) · American Journal of Political Science · doi:10.1111/ajps.12103 · Cited by 1,952
(2009) · Cited by 1,900
Model Selection: An Integral Part of Inference
Buckland, S. T., Burnham, K. P., & Augustin, N. H. (1997) · Biometrics · doi:10.2307/2533961 · Cited by 1,717
Extracting Policy Positions from Political Texts Using Words as Data
LAVER, M., BENOIT, K., & GARRY, J. (2003) · American Political Science Review · doi:10.1017/s0003055403000698 · Cited by 1,377
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963
Evaluation methods for topic models
Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553515 · Cited by 863
Technical terminology: some linguistic properties and an algorithm for identification in text
Justeson, J. S., & Katz, S. M. (1995) · Natural Language Engineering · doi:10.1017/s1351324900000048 · Cited by 820
A Method of Automated Nonparametric Content Analysis for Social Science
Hopkins, D. J., & King, G. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00428.x · Cited by 765
A Scaling Model for Estimating Time‐Series Party Positions from Texts
Slapin, J. B., & Proksch, S. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00338.x · Cited by 684
How to analyze political attention with minimal assumptions and costs
Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00427.x · Cited by 671
Grimmer, J. (2010) · Political Analysis · doi:10.1093/pan/mpp034 · Cited by 550
Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008) · Political Analysis · doi:10.1093/pan/mpn018 · Cited by 542 · cited 1×
“This may be because the output of those models does not easily lend itself to answering substantive questions (see, e.g. Monroe, Colaresi and Quinn, 2008) or perhaps because the assumptions underlying those techniques are both consequential, and non-trivial to fully understand (see, e.g. Lowe,…”
BackgroundPosition Taking in European Parliament Speeches
Proksch, S.-O., & Slapin, J. B. (2010) · British Journal of Political Science · doi:10.1017/s0007123409990299 · Cited by 254
Multiple Comparisons in Induction Algorithms
Jensen, D. D., & Cohen, P. R. (2000) · Machine Learning · doi:10.1023/a:1007631014630 · Cited by 241
Computer‐Assisted Keyword and Document Set Discovery from Unstructured Text
King, G., Lam, P., & Roberts, M. E. (2017) · American Journal of Political Science · doi:10.1111/ajps.12291 · Cited by 229
General purpose computer-assisted clustering and conceptualization
Grimmer, J., & King, G. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1018067108 · Cited by 179
Measuring Political Positions from Legislative Speech
Lauderdale, B. E., & Herzog, A. (2016) · Political Analysis · doi:10.1093/pan/mpw017 · Cited by 173
Language and Ideology in Congress
Diermeier, D., Godbout, J.-F., Yu, B., & Kaufmann, S. (2012) · British Journal of Political Science · doi:10.1017/s0007123411000160 · Cited by 167
Lowe, W. (2008) · Political Analysis · doi:10.1093/pan/mpn004 · Cited by 166 · cited 1×
“…may be because the output of those models does not easily lend itself to answering substantive questions (see, e.g. Monroe, Colaresi and Quinn, 2008) or perhaps because the assumptions underlying those techniques are both consequential, and non-trivial to fully understand (see, e.g. Lowe, 2008).”
BackgroundFrom Pork to Policy: The Rise of Programmatic Campaigning in Japanese Elections
Catalinac, A. L. (2015) · The Journal of Politics · doi:10.1086/683073 · Cited by 137
U.S. Treaty Making with American Indians: Institutional Change and Relative Power, 1784–1911
Spirling, A. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00558.x · Cited by 98
Self-citationValidating Estimates of Latent Traits from Textual Data Using Human Judgment as a Benchmark
Lowe, W., & Benoit, K. R. (2013) · Political Analysis · doi:10.1093/pan/mpt002 · Cited by 81
D'Orazio, V. J., Landis, S. T., Palmer, G., & Schrodt, P. A. (2014) · Political Analysis · doi:10.1093/pan/mpt030 · Cited by 64
Bag of What? Simple Noun Phrase Extraction for Text Analysis
Handler, A., Denny, M. J., Wallach, H., & O'Connor, B. (2016) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/w16-5615 · Cited by 58
Self-citationTextual Predictors of Bill Survival in Congressional Committees
Yano, T., Smith, N. A., & Wilkerson, J. (2018) · doi:10.1184/r1/6473729 · Cited by 49
The Reordering of British Politics : Politics after Thatcher
Kavanagh, D. (1997) · Cited by 32
Driving support: workers, PACs, and congressional support of the auto industry
Moore, R. T., Powell, E. N., & Reeves, A. (2013) · Business and Politics · doi:10.1515/bap-2013-0005 · Cited by 20
Preregistration of Studies and Mock Reports
Gelman, A. (2013) · Political Analysis · doi:10.1093/pan/mps032 · Cited by 20
Cited by
Gentzkow, M., Kelly, B., & Taddy, M. (2019) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,236
Smart literature review: a practical topic modelling approach to exploratory literature review
Asmussen, C. B., & Møller, C. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0255-7 · Cited by 455
Rhetoric and frame analysis of ExxonMobil's climate change communications
Supran, G., & Oreskes, N. (2021) · One Earth · doi:10.1016/j.oneear.2021.04.014 · Cited by 374
Machine Learning for Social Science: An Agnostic Approach
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2021) · Annual Review of Political Science · doi:10.1146/annurev-polisci-053119-015921 · Cited by 323
Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300
Metadata
- Authors
- Denny, M. J., & Spirling, A.
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-03-19
- Volume
- 26
- Issue
- 2
- Pages
- 168-189
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W3125733373
- Cited by
- 463
- References
- 42
- FWCI
- 158.38
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.