The DOI matches its Crossref record.
A stop list for general text
Abstract
A stop list , or negative dictionary is a device used in automatic indexing to filter out words that would make poor index terms. Traditionally stop lists are supposed to have included only the most frequently occurring words. In practice, however, stop lists have tended to include infrequently occurring words, and have not included many frequently occurring words. Infrequently occurring words seem to have been included because stop list compilers have not, for whatever reason, consulted empirical studies of word frequencies. Frequently occurring words seem to have been left out for the same reason, and also because many of them might still be important as index terms.This paper reports an exercise in generating a stop list for general text based on the Brown corpus of 1,014,000 words drawn from a broad range of literature in English. We start with a list of tokens occurring more than 300 times in the Brown corpus. From this list of 278 words, 32 are culled on the grounds that they are too important as potential index terms. Twenty-six words are then added to the list in the belief that they may occur very frequently in certain kinds of literature. Finally, 149 words are added to the list because the finite state machine based filter in which this list is intended to be used is able to filter them at almost no cost. The final product is a list of 421 stop words that should be maximally efficient and effective in filtering the most frequently occurring and semantically neutral words in general literature in English.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Automatic Keyword Extraction from Individual Documents
Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010) · Text Mining · doi:10.1002/9780470689646.ch1 · Cited by 1,095
Translation techniques in cross-language information retrieval
Zhou, D., Truran, M., Brailsford, T., Wade, V., & Ashman, H. (2012) · ACM Computing Surveys · doi:10.1145/2379776.2379777 · Cited by 698
What Drives Media Slant? Evidence from U.S. Daily Newspapers
Gentzkow, M., & Shapiro, J. (2006) · National Bureau of Economic Research · doi:10.3386/w12707 · Cited by 561
What Drives Media Slant? Evidence from U.S. Daily Newspapers
Gentzkow, M. A., & Shapiro, J. M. (2006) · SSRN Electronic Journal · doi:10.2139/ssrn.947640 · Cited by 434
The influence of preprocessing on text classification using a bag-of-words representation
HaCohen-Kerner, Y., Miller, D., & Yigal, Y. (2020) · PLOS ONE · doi:10.1371/journal.pone.0232525 · Cited by 266
Metadata
- Authors
- Fox, C.
- Venue
- ACM SIGIR Forum
- Published
- 1989-09
- Volume
- 24
- Issue
- 1-2
- Pages
- 19-21
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W2029115643
- Cited by
- 325
- References
- 0
- FWCI
- 1.49
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A stop list for general text. https://app.cochat.ai/papers/10.1145/378881.378888
- The paper
- Fox, C. (1989). A stop list for general text. ACM SIGIR Forum. https://doi.org/10.1145/378881.378888
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.