The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
IR evaluation methods for retrieving highly relevant documents
Abstract
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modem large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Practical Nonparametric Statistics
Groggel, D. J. (2000) · Technometrics · doi:10.1080/00401706.2000.10486067 · Cited by 11,041
Practical Nonparametric Statistics (2nd ed).
Groeneveld, R. A., & Conover, W. J. (1981) · Journal of the American Statistical Association · doi:10.2307/2287607 · Cited by 1,849
Practical Nonparametric Statistics.
Haigh, J., & Conover, W. J. (1981) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981807 · Cited by 1,503
An evaluation of retrieval effectiveness for a full-text document-retrieval system
Blair, D. C., & Maron, M. E. (1985) · Communications of the ACM · doi:10.1145/3166.3197 · Cited by 656
Inference Networks for Document Retrieval
Turtle, H. R., & Croft, W. B. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130361 · Cited by 423
A study of information seeking and retrieving. I. Background and methodology
Saracevic, T., Kantor, P., Chamis, A. Y., & Trivison, D. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198805)39:3<161::aid-asi2>3.0.co;2-0 · Cited by 239
Practical Nonparametric Statistics (2nd ed.)
Obremski, T. E. (1981) · Technometrics · doi:10.1080/00401706.1981.10487689 · Cited by 189
Measures of relative relevance and ranked half-life
Borlund, P., & Ingwersen, P. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.291019 · Cited by 96
The impact of query structure and query expansion on retrieval performance
Kekäläinen, J., & Järvelin, K. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.290978 · Cited by 96
Self-citation(2015) · Langaa RPCIG eBooks · doi:10.2307/j.ctvh9vz35.7 · Cited by 73
Cited by
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Self-citationLearning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
Learning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960
Foundations of Machine Learning
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018) · doi:10.7551/mitpress/10290.001.0001 · Cited by 1,122
Agrawal, R., Gollapudi, S., Halverson, A., & Ieong, S. (2009) · Proceedings of the Second ACM International Conference on Web Search and Data Mining · doi:10.1145/1498759.1498766 · Cited by 1,020
Metadata
- Authors
- Järvelin, K., & Kekäläinen, J.
- Venue
- ACM SIGIR Forum
- Published
- 2017-08-02
- Volume
- 51
- Issue
- 2
- Pages
- 243-250
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W1985554184
- Cited by
- 1,444
- References
- 17
- FWCI
- 125.31
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: IR evaluation methods for retrieving highly relevant documents. https://app.cochat.ai/papers/10.1145/3130348.3130374
- The paper
- Järvelin, K., & Kekäläinen, J. (2017). IR evaluation methods for retrieving highly relevant documents. ACM SIGIR Forum. https://doi.org/10.1145/3130348.3130374
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
