The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
IR evaluation methods for retrieving highly relevant documents
Open accessAbstract
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query1) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Practical Nonparametric Statistics.
Haigh, J., & Conover, W. J. (1981) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981807 · Cited by 1,503
An evaluation of retrieval effectiveness for a full-text document-retrieval system
Blair, D. C., & Maron, M. E. (1985) · Communications of the ACM · doi:10.1145/3166.3197 · Cited by 656
A study of information seeking and retrieving. I. Background and methodology
Saracevic, T., Kantor, P., Chamis, A. Y., & Trivison, D. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198805)39:3<161::aid-asi2>3.0.co;2-0 · Cited by 239
Practical Nonparametric Statistics (2nd ed.)
Obremski, T. E. (1981) · Technometrics · doi:10.1080/00401706.1981.10487689 · Cited by 189
Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Borlund, P., & Ingwersen, P. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.291019 · Cited by 115
The impact of query structure and query expansion on retrieval performance
Kekäläinen, J., & Järvelin, K. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.290978 · Cited by 96
Self-citationLosee, R. M. (1998) · The Kluwer international series on information retrieval · doi:10.1007/978-1-4615-5705-0 · Cited by 42
Robertson, S. E., & Belkin, N. J. (1978) · Journal of Documentation · doi:10.1108/eb026654 · Cited by 35
Kekäläinen, J., & Järvelin, K. (2000) · Information Retrieval · doi:10.1023/a:1009983401464 · Cited by 27
Self-citation
Cited by
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
Learning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960
“In ranking performance evaluation, we adopted two common IR evaluation measures: Normalized Discounted Cumulative Gain (NDCG) (Jarvelin & Kekanainen, 2000) and Mean Average Precision (MAP)(Baeza-Yates & Ribeiro-Neto, 1999).”
MethodLearning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Shilane, P., Min, P., Kazhdan, M., & Funkhouser, T. (2004) · Proceedings Shape Modeling Applications, 2004. · doi:10.1109/smi.2004.1314504 · Cited by 1,419
Improving web search ranking by incorporating user behavior information
Agichtein, E., Brill, E., & Dumais, S. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148177 · Cited by 1,041
Metadata
- Authors
- Järvelin, K., & Kekäläinen, J.
- Venue
- Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2000-07
- Pages
- 41-48
- OpenAlex
- W4251560691
- Cited by
- 967
- References
- 9
- FWCI
- 5.93
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.