Cumulated gain-based evaluation of IR techniques
Abstract
Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
nDCG is a commonly used relevance metric in the literature.
“This is a commonly used relevance metric in the literature.”
From Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search (2022) · cites this paper as [7]
The design of search result pages assumes that people scan from the top and that attention declines with rank, and discounted cumulative gain encodes the same assumption in evaluation.
“The design assumes that people scan from the top and that attention declines with rank, and discounted cumulative gain encodes the same assumption in evaluation [27].”
From Agentic Memory Management Is the Next Frontier of Information Retrieval (2026) · cites this paper as [27]
Cites
How reliable are the results of large-scale information retrieval experiments?
Zobel, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291014 · Cited by 579 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation
Goutte, C., & Gaussier, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_25 · Cited by 2,285 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 368 · see citing works
Measuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378 · see citing works
Large Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225 · see citing works
Dense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211 · see citing works
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
The LambdaLoss Framework for Ranking Metric Optimization
Wang, X., Li, C., Golbandi, N., Bendersky, M., & Najork, M. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271784 · Cited by 136 · see citing works
Predicting User Satisfaction with Intelligent Assistants
Kiseleva, J., Williams, K., Hassan Awadallah, A., Crook, A. C., Zitouni, I., & Anastasakos, T. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911521 · Cited by 108 · see citing works
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
Quantifying the trustworthiness of social media content
Moturu, S. T., & Liu, H. (2011) · Distributed and Parallel Databases · doi:10.1007/s10619-010-7077-0 · Cited by 56 · see citing works
GastroBot: a Chinese gastrointestinal disease chatbot based on the retrieval-augmented generation
Zhou, Q., Liu, C., Duan, Y., Sun, K., Li, Y., Kan, H., Gu, Z., Shu, J., & Hu, J. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1392555 · Cited by 44 · see citing works
ReuseKNN: Neighborhood Reuse for Differentially Private KNN-Based Recommendations
Müllner, P., Lex, E., Schedl, M., & Kowald, D. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3608481 · Cited by 27 · see citing works
A Method to Learn Embedding of a Probabilistic Medical Knowledge Graph: Algorithm Development
Li, L., Wang, P., Wang, Y., Wang, S., Yan, J., Jiang, J., Tang, B., Wang, C., & Liu, Y. (2020) · JMIR Medical Informatics · doi:10.2196/17645 · Cited by 25 · see citing works
DeepFT: Fault-Tolerant Edge Computing using a Self-Supervised Deep Surrogate Model
Tuli, S., Casale, G., Cherkasova, L., & Jennings, N. R. (2023) · IEEE INFOCOM 2023 - IEEE Conference on Computer Communications · doi:10.1109/infocom53939.2023.10229049 · Cited by 23 · see citing works
Enabling Hyper-Personalisation: Automated Ad Creative Generation and Ranking for Fashion e-Commerce
Vempati, S., Malayil, K. T., Sruthi, V., & Sandeep, R. (2020) · Lecture Notes in Social Networks · doi:10.1007/978-3-030-55218-3_2 · Cited by 24 · see citing works
DRAGON: Decentralized Fault Tolerance in Edge Federations
Tuli, S., Casale, G., & Jennings, N. R. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3199886 · Cited by 21 · see citing works
Towards Safety and Sustainability: Designing Local Recommendations for Post-pandemic World
Patro, G. K., Chakraborty, A., Banerjee, A., & Ganguly, N. (2020) · Fourteenth ACM Conference on Recommender Systems · doi:10.1145/3383313.3412251 · Cited by 21
Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search
Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A., & Subbian, K. (2022) · arXiv · doi:10.48550/arxiv.2206.06588 · Cited by 17 · see citing works
MRR: an unsupervised algorithm to rank reviews by relevance
Woloszyn, V., dos Santos, H. D. P., Wives, L. K., & Becker, K. (2017) · Proceedings of the International Conference on Web Intelligence · doi:10.1145/3106426.3106444 · Cited by 9 · see citing works
Personalized Chit-Chat Generation for Recommendation Using External Chat Corpora
Chen, C., Wang, X., Yi, X., Wu, F., Xie, X., & Yan, R. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539215 · Cited by 5 · see citing works
Gilani, K., Verket, M., Peters, C., Dumontier, M., Rocca, H.-P. B.-L., & Urovi, V. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110745 · Cited by 5 · see citing works
Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks
Panthi, S., & Abdelfattah, R. (2026) · arXiv · doi:10.48550/arxiv.2605.24060
“The closest analogy is classical IR work on evaluation sensitivity (Bailey et al., 2008; Yang et al., 2019; Craswell et al., 2020; Voorhees, 2001; Voorhees and Harman, 2005; Manning et al., 2008; Järvelin and Käkäläinen, 2002).”
PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
- Authors
- Järvelin, K., & Kekäläinen, J.
- Affiliations
- University of Tampere (FI)
- Venue
- ACM Transactions on Information Systems
- Published
- 2002-10
- Volume
- 20
- Issue
- 4
- Pages
- 422-446
- ISSN
- 1046-8188 (ISSN-L)
- 1558-2868
- OpenAlex
- W2069870183
- Cited by
- 4,841
- References
- 1
- Citations by year
- 2026: 400 · 2025: 305 · 2024: 269 · 2023: 273 · 2022: 256 · 2021: 286 · 2020: 259 · 2019: 290 · 2018: 245 · 2017: 219 · 2016: 267 · 2015: 236 · 2014: 273 · 2013: 264 · 2012: 246
- FWCI
- 20.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Information Systems (publishedVersion)
- Funders
- McGill University
- Tampereen Yliopisto
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Algorithms and Data Compression
- Keywords
- information retrieval evaluation, ranked retrieval, relevance scores, discount factor, effectiveness measures, TREC-7
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
