The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
IR evaluation methods for retrieving highly relevant documents
Open accessAbstract
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query1) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Practical Nonparametric Statistics.
Haigh, J., & Conover, W. J. (1981) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981807 · Cited by 1,503
An evaluation of retrieval effectiveness for a full-text document-retrieval system
Blair, D. C., & Maron, M. E. (1985) · Communications of the ACM · doi:10.1145/3166.3197 · Cited by 656
A study of information seeking and retrieving. I. Background and methodology
Saracevic, T., Kantor, P., Chamis, A. Y., & Trivison, D. (1988) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(198805)39:3<161::aid-asi2>3.0.co;2-0 · Cited by 239
Practical Nonparametric Statistics (2nd ed.)
Obremski, T. E. (1981) · Technometrics · doi:10.1080/00401706.1981.10487689 · Cited by 189
Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Borlund, P., & Ingwersen, P. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.291019 · Cited by 115
The impact of query structure and query expansion on retrieval performance
Kekäläinen, J., & Järvelin, K. (1998) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/290941.290978 · Cited by 96
Self-citationLosee, R. M. (1998) · The Kluwer international series on information retrieval · doi:10.1007/978-1-4615-5705-0 · Cited by 42
Robertson, S. E., & Belkin, N. J. (1978) · Journal of Documentation · doi:10.1108/eb026654 · Cited by 35
Kekäläinen, J., & Järvelin, K. (2000) · Information Retrieval · doi:10.1023/a:1009983401464 · Cited by 27
Self-citation
Cited by
Learning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
Learning to rank: from pairwise approach to listwise approach
Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., & Li, H. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273513 · Cited by 1,960
“In ranking performance evaluation, we adopted two common IR evaluation measures: Normalized Discounted Cumulative Gain (NDCG) (Jarvelin & Kekanainen, 2000) and Mean Average Precision (MAP)(Baeza-Yates & Ribeiro-Neto, 1999).”
MethodLearning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Shilane, P., Min, P., Kazhdan, M., & Funkhouser, T. (2004) · Proceedings Shape Modeling Applications, 2004. · doi:10.1109/smi.2004.1314504 · Cited by 1,419
Improving web search ranking by incorporating user behavior information
Agichtein, E., Brill, E., & Dumais, S. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148177 · Cited by 1,041
Agrawal, R., Gollapudi, S., Halverson, A., & Ieong, S. (2009) · Proceedings of the Second ACM International Conference on Web Search and Data Mining · doi:10.1145/1498759.1498766 · Cited by 1,020
Personalizing search via automated analysis of interests and activities
Teevan, J., Dumais, S. T., & Horvitz, E. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076111 · Cited by 800
AdaRank: a boosting algorithm for information retrieval
Xu, J., & Li, H. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277809 · Cited by 775
Detecting product review spammers using rating behaviors
Lim, E.-P., Nguyen, V.-A., Jindal, N., Liu, B., & Lauw, H. W. (2010) · Proceedings of the 19th ACM international conference on Information and knowledge management · doi:10.1145/1871437.1871557 · Cited by 715
A support vector method for optimizing average precision
Yue, Y., Finley, T., Radlinski, F., & Joachims, T. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277790 · Cited by 704
Learning semantic representations using convolutional neural networks for web search
Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G. (2014) · Proceedings of the 23rd International Conference on World Wide Web · doi:10.1145/2567948.2577348 · Cited by 670
A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval
Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G. (2014) · Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management · doi:10.1145/2661829.2661935 · Cited by 661
Latent aspect rating analysis on review text data: a rating regression approach
Wang, H., Lu, Y., & Zhai, C. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835903 · Cited by 650
Palangi, H., Deng, L., Shen, Y., Gao, J., He, X., Chen, J., Song, X., & Ward, R. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2520371 · Cited by 618
Three-dimensional shape searching: state-of-the-art review and future trends
Iyer, N., Jayanti, S., Lou, K., Kalyanaraman, Y., & Ramani, K. (2005) · Computer-Aided Design · doi:10.1016/j.cad.2004.07.002 · Cited by 603
Adapting ranking SVM to document retrieval
Cao, Y., Xu, J., Liu, T.-Y., Li, H., Huang, Y., & Hon, H.-W. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148205 · Cited by 569
Adapting boosting for information retrieval measures
Wu, Q., Burges, C. J. C., Svore, K. M., & Gao, J. (2010) · Information Retrieval · doi:10.1007/s10791-009-9112-1 · Cited by 567
Deep semantic ranking based hashing for multi-label image retrieval
Zhao, F., Huang, Y., Wang, L., & Tan, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298763 · Cited by 564
Beyond Independent Relevance: Methods and Evaluation Metrics for Subtopic Retrieval
Zhai, C., Cohen, W. W., & Lafferty, J. (2015) · ACM SIGIR Forum · doi:10.1145/2795403.2795405 · Cited by 558
Optimizing web search using social annotations
Bao, S., Xue, G., Wu, X., Yu, Y., Fei, B., & Su, Z. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242640 · Cited by 546
Sequential Recommendation with User Memory Networks
Chen, X., Xu, H., Zhang, Y., Tang, J., Cao, Y., Qin, Z., & Zha, H. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159668 · Cited by 498
The concept of relevance in IR
Borlund, P. (2003) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10286 · Cited by 482
Learning Hierarchical Representation Model for NextBasket Recommendation
Wang, P., Guo, J., Lan, Y., Xu, J., Wan, S., & Cheng, X. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767694 · Cited by 411
Liu, X., Gao, J., He, X., Deng, L., Duh, K., & Wang, Y.-Y. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1092 · Cited by 395
End-to-End Training of Object Class Detectors for Mean Average Precision
Henderson, P., & Ferrari, V. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54193-8_13 · Cited by 373
Information retrieval system evaluation: effort, sensitivity, and reliability
Sanderson, M., & Zobel, J. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076064 · Cited by 369
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405
Beyond independent relevance: methods and evaluation metrics for subtopic retrieval
Zhai, C. X., Cohen, W. W., & Lafferty, J. (2003) · Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval · doi:10.1145/860435.860440 · Cited by 366
A Short Introduction to Learning to Rank
LI, H. (2011) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e94.d.1854 · Cited by 348
Controllable Multi-Interest Framework for Recommendation
Cen, Y., Zhang, J., Zou, X., Zhou, C., Yang, H., & Tang, J. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403344 · Cited by 338
Personalized query expansion for the web
Chirita, P. A., Firan, C. S., & Nejdl, W. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277746 · Cited by 290
Chen, X., Chen, H., Xu, H., Zhang, Y., Cao, Y., Qin, Z., & Zha, H. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331254 · Cited by 284
Generalized distances between rankings
Kumar, R., & Vassilvitskii, S. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772749 · Cited by 273
A Survey of Graph Neural Networks for Social Recommender Systems
Sharma, K., Lee, Y.-C., Nambi, S., Salian, A., Shah, S., Kim, S.-W., & Kumar, S. (2024) · ACM Computing Surveys · doi:10.1145/3661821 · Cited by 268
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155
Turning Clicks into Purchases: Revenue Optimization for Product Search in E-Commerce
Wu, L., Hu, D., Hong, L., & Liu, H. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3209993 · Cited by 85
GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation
Zhou, Q., Liu, C., Duan, Y., Sun, K., Li, Y., Kan, H., Gu, Z., Shu, J., & Hu, J. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1392555 · Cited by 44
Freshness matters: in flowers, food, and web authority
Dai, N., & Davison, B. D. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835471 · Cited by 36
Liu, Y., Moretti, R., Wang, Y., Dong, H., Yan, B., Bodenheimer, B., Derr, T., & Meiler, J. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00822 · Cited by 13
Metadata
- Authors
- Järvelin, K., & Kekäläinen, J.
- Venue
- Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2000-07
- Pages
- 41-48
- OpenAlex
- W4251560691
- Cited by
- 967
- References
- 9
- FWCI
- 5.93
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.