The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
The automatic identification of stop words
Abstract
A stop word may be identified as a word that has the same likehhood of occurring in those documents not relevant to a query as in those documents relevant to the query. In this paper we show how the concept of relevance may be replaced by the condition of being highly rated by a similarity measure. Thus it becomes possible to identify the stop words in a cullectmn by automated statistical testing. We describe the nature of the statistical test as it is realized with a vector retrieval methodology based on the cosine coefficient of document-document similarity. As an example, this tech nique is then applied to a large MEDLINE " subset in the area of biotechnology. The initial processing of this datahase involves a 310 word stop list of common non-content terms. Our technique is then applied and 75% of the remaining terms are identified as stop words. We compare retrieval with and without the removal of these stop words and find that of the top twenty documents retrieved in response to a random query document. seventeen of these are the same on the average for the two methods We also examine the differences and conclude that where the user prefers one method over the other, the new method with the reduced term set is favored about three times out of four.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to Modern Information Retrieval
Salton, G., & McGill, M. J. (1983) · Virtual Defense Library (Ministerio de Defensa) · Cited by 11,392
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
A vector space model for automatic indexing
Salton, G., Wong, A., & Yang, C. S. (1975) · Communications of the ACM · doi:10.1145/361219.361220 · Cited by 7,521
Introduction to modern information retrieval
Dillon, M. (1983) · Information Processing & Management · doi:10.1016/0306-4573(83)90062-6 · Cited by 5,909
A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
SPARCK JONES, K. (1972) · Journal of Documentation · doi:10.1108/eb026526 · Cited by 4,505
Salton, G. (1988) · Cited by 1,587
Bookstein, A. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90016-u · Cited by 699
Implementation of the SMART Information Retrieval System
Buckley, C. (1985) · eCommons (Cornell University) · Cited by 289
Optimization of inverted vector searches
Buckley, C., & Lewit, A. F. (1985) · Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '85 · doi:10.1145/253495.253515 · Cited by 216
Historical note: Information retrieval and the future of an illusion
Swanson, D. R. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198803)39:2<92::aid-asi4>3.0.co;2-p · Cited by 170
Cited by
Temporal patterns of happiness and information in a global social network: hedonometrics and Twitter
Dodds, P. S., Harris, K. D., Kloumann, I. M., Bliss, C. A., & Danforth, C. M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026752 · Cited by 809
“In varying word sets by excluding stop words [58], we can systematically explore families of happiness metrics.”
BackgroundA Survey of Text Clustering Algorithms
Aggarwal, C. C., & Zhai, C. (2012) · Mining Text Data · doi:10.1007/978-1-4614-3223-4_4 · Cited by 581
Cronen-Townsend, S., & Croft, W. B. (2002) · Proceedings of the second international conference on Human Language Technology Research - · doi:10.3115/1289189.1289266 · Cited by 132
“Another focus of research attention started with the automatic choice of index terms[1] and led to automatic identification of stop words[19] and content-bearing terms[8].”
BackgroundS3BD: Secure semantic search over encrypted big data in the cloud
Woodworth, J. W., & Amini Salehi, M. (2019) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.5050 · Cited by 15
Metadata
- Authors
- Wilbur, W. J., & Sirotkin, K.
- Venue
- Journal of Information Science
- Publisher
- SAGE Publications
- Published
- 1992-02
- Volume
- 18
- Issue
- 1
- Pages
- 45-55
- ISSN
- 0165-5515 (ISSN-L)
- 1741-6485
- OpenAlex
- W2121537574
- Cited by
- 295
- References
- 11
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The automatic identification of stop words. https://app.cochat.ai/papers/10.1177/016555159201800106
- The paper
- Wilbur, W. J., & Sirotkin, K. (1992). The automatic identification of stop words. Journal of Information Science. https://doi.org/10.1177/016555159201800106
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.