The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Novelty and diversity in information retrieval evaluation
Abstract
Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Learning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408
Cited by
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
Kaminskas, M., & Bridge, D. (2017) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2926720 · Cited by 465
Algorithmic Effects on the Diversity of Consumption on Spotify
Anderson, A., Maystre, L., Anderson, I., Mehrotra, R., & Lalmas, M. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380281 · Cited by 206
“The information retrieval community has been investigating diversity metrics in a number of domains for decades (e.g. see [5, 25]), and more recently, diversity measures have been used to quantify notions of fairness in online platforms [18].”
BackgroundUnifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161
Diversity in Big Data: A Review
Drosou, M., Jagadish, H. V., Pitoura, E., & Stoyanovich, J. (2017) · Big Data · doi:10.1089/big.2016.0054 · Cited by 140
User Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000098 · Cited by 52
Fairness-aware Federated Matrix Factorization
Liu, S., Ge, Y., Xu, S., Zhang, Y., & Marian, A. (2022) · Proceedings of the 16th ACM Conference on Recommender Systems · doi:10.1145/3523227.3546771 · Cited by 43
Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767771 · Cited by 38
Self-citationA reinforcement learning framework for relevance feedback
Montazeralghaem, A., Zamani, H., & Allan, J. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401099 · Cited by 36
FAIR : Fairness‐aware information retrieval evaluation
Gao, R., Ge, Y., & Shah, C. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24648 · Cited by 19
Coverage-based search result diversification
Zheng, W., Wang, X., Fang, H., & Cheng, H. (2012) · Information Retrieval · doi:10.1007/s10791-011-9178-4 · Cited by 16
Recency ranking by diversification of result set
Styskin, A., Romanenko, F., Vorobyev, F., & Serdyukov, P. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063862 · Cited by 14
An Instruction-Response Perspective on Large Language Models in Information Retrieval Tasks
Joho, H., & Jose, J. M. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730346 · Cited by 3
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Thakur, N., Lin, J., Havens, S., Carbin, M., Khattab, O., & Drozdov, A. (2025) · arXiv · doi:10.48550/arxiv.2504.13128
CoverageBench: Evaluating Information Coverage across Tasks and Domains
Samuel, S., Yates, A., Lawrie, D., Soboroff, I., Adriaanse, T., Van Durme, B., & Yang, E. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808625
Metadata
- Authors
- Clarke, C. L. A., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Büttcher, S., & MacKinnon, I.
- Venue
- Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2008-07-20
- Pages
- 659-666
- OpenAlex
- W2132314908
- Cited by
- 962
- References
- 3
- FWCI
- 42.76
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
