The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
A Language Modeling Approach to Information Retrieval
Open accessAbstract
In today's world, there is no shortage of information. However, for a specific information need, only a small subset of all of the available information will be useful. The field of information retrieval (IR) is the study of methods to provide users with that small subset of information relevant to their needs and to do so in a timely fashion. Information sources can take many forms, but this thesis will focus on text based information systems and investigate problems germane to the retrieval of written natural language documents. Central to these problems is the notion of "topic." In other words, what are documents about? However, topics depend on the semantics of documents and retrieval systems are not endowed with knowledge of the semantics of natural language. The approach taken in this thesis will be to make use of probabilistic language models to investigate text based information retrieval and related problems. One such problem is the prediction of topic shifts in text, the topic segmentation problem. It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection. Two complementary sets of features are studied individually and then combined into a single language model. The language modeling approach allows this problem to be approached in a principled way without complex semantic modeling. Next, the problem of document retrieval in response to a user query will be investigated. Models of document indexing and document retrieval have been extensively studied over the past three decades. The integration of these two classes of models has been the goal of several researchers but it is a very difficult problem. Much of the reason for this is that the indexing component requires inferences as to the semantics of documents. Instead, an approach to retrieval based on probabilistic language modeling will be presented. Models are estimated for each document individually. The approach to modeling is non-parametric and integrates the entire retrieval process into a single model. One advantage of this approach is that collection statistics, which are used heuristically for the assignment of concept probabilities in other probabilistic models, are used directly in the estimation of language model probabilities in this approach. The language modeling approach has been implemented and tested empirically and performs very well on standard test collections and query sets. In order to improve retrieval effectiveness, IR systems use additional techniques such as relevance feedback, unsupervised query expansion and structured queries. These and other techniques are discussed in terms of the language modeling approach and empirical results are given for several of the techniques developed. These results provide further proof of concept for the use of language models for retrieval tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Handbook of Mathematical Functions
McQuarrie, D. A. (1966) · American Journal of Physics · doi:10.1119/1.1972842 · Cited by 40,433
Density Estimation for Statistics and Data Analysis
Silverman, B. W. (2018) · Routledge eBooks · doi:10.1201/9781315140919 · Cited by 15,796
On Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,585
Statistical Analysis of Finite Mixture Distributions.
Lindsay, B. G., Titterington, D. M., Smith, A. F. M., & Makov, U. E. (1987) · Journal of the American Statistical Association · doi:10.2307/2289504 · Cited by 2,905
Bhunia, C. T. (1997) · IETE Journal of Education · doi:10.1080/09747338.1997.11415666 · Cited by 2,701
Relevance weighting of search terms
Robertson, S. E., & Jones, K. S. (1976) · Journal of the American Society for Information Science · doi:10.1002/asi.4630270302 · Cited by 2,081
(1964) · doi:10.1115/1.3625776 · Cited by 1,796
Salton, G. (1988) · Cited by 1,587
Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval
Robertson, S. E., & Walker, S. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_24 · Cited by 1,477
Greenwood, M., & Yule, G. U. (1920) · Journal of the Royal Statistical Society · doi:10.2307/2341080 · Cited by 833
Bookstein, A. (1990) · Information Processing & Management · doi:10.1016/0306-4573(90)90016-u · Cited by 699
USING PROBABILISTIC MODELS OF DOCUMENT RETRIEVAL WITHOUT RELEVANCE INFORMATION
CROFT, W. B., & HARPER, D. J. (1979) · Journal of Documentation · doi:10.1108/eb026683 · Cited by 433
Self-citationPonte, J. M., & Croft, W. B. (1997) · Lecture Notes in Computer Science · doi:10.1007/bfb0026725 · Cited by 213
Self-citationProbabilistic models for automatic indexing
Bookstein, A., & Swanson, D. R. (1974) · Journal of the American Society for Information Science · doi:10.1002/asi.4630250505 · Cited by 181
Models for retrieval with probabilistic indexing
Fuhr, N. (1989) · Information Processing & Management · doi:10.1016/0306-4573(89)90091-5 · Cited by 177
A probabilistic approach to automatic keyword indexing
Harter, S. P. (1974) · Journal of the American Society for Information Science and Technology · Cited by 147
A new method of weighting query terms for ad-hoc retrieval
Kwok, K. L. (1996) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/243199.243266 · Cited by 82
Efficient probabilistic inference for text retrieval
Turtle, H. R., & Croft, W. B. (1991) · Cited by 74
Self-citationConstruction of Improved Estimators in Multiparameter Estimation for Discrete Exponential Families
Ghosh, M., Hwang, J. T., & Tsui, K. (1983) · The Annals of Statistics · doi:10.1214/aos/1176346143 · Cited by 60
A probability distribution model for information retrieval
Wong, S. K. M., & Yao, Y. Y. (1989) · Information Processing & Management · doi:10.1016/0306-4573(89)90090-3 · Cited by 44
Modelling documents with multiple poisson distributions
Margulis, E. L. (1993) · Information Processing & Management · doi:10.1016/0306-4573(93)90005-x · Cited by 20
Cited by
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,720
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Learning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Understanding inverse document frequency: on theoretical arguments for IDF
Robertson, S. (2004) · Journal of Documentation · doi:10.1108/00220410410560582 · Cited by 1,541
A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval
Zhai, C., & Lafferty, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130377 · Cited by 1,537
Metadata
- Authors
- Ponte, J. M., & Croft, W. B.
- Venue
- ACM SIGIR Forum
- Published
- 2017-08-02
- Volume
- 51
- Issue
- 2
- Pages
- 202-208
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W2093390569
- Cited by
- 2,503
- References
- 21
- FWCI
- 174.22
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.